Nova Resource:Tools/SAL

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.47.0-wmf.10</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Nova Resource:Tools/SAL</title>
    <ns>498</ns>
    <id>3086</id>
    <revision>
      <id>2435495</id>
      <parentid>2435094</parentid>
      <timestamp>2026-07-11T17:34:33Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>dcaro: rebooting haproxy-8</comment>
      <origin>2435495</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="787222" sha1="r0w1gx41v8hto8uihycvcb1wvald549" xml:space="preserve">=== 2026-07-11 ===
* 17:34 dcaro: rebooting haproxy-8

=== 2026-07-09 ===
* 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 12:53 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:27 taavi: rebooting bastions for kernel upgrades
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-08 ===
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-07 ===
* 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-06 ===
* 17:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-02 ===
* 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])
* 10:18 taavi: deploy latest striker with [[phab:T430120|T430120]] [[phab:T409341|T409341]] fixes
* 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-30 ===
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-29 ===
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-25 ===
* 17:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 08:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]])
* 08:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]])

=== 2026-06-24 ===
* 16:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2026-06-23 ===
* 08:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-22 ===
* 10:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-18 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-15 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 12:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]])
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]])
* 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:45 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]])
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]])
* 11:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]])
* 09:53 godog: delete tools-checker puppet prefix - [[phab:T313030|T313030]]
* 09:52 godog: delete tools-checker-5 - [[phab:T313030|T313030]]

=== 2026-06-11 ===
* 13:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]])
* 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]])

=== 2026-06-09 ===
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 11:01 taavi: starting Kyverno upgrade [[phab:T427928|T427928]]

=== 2026-06-04 ===
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway
* 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 12:15 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway
* 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:06 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7
* 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4
* 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4
* 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4
* 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4
* 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4
* 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4
* 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:02 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4
* 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry

=== 2026-06-02 ===
* 15:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 10:33 taavi: migrate acme-chief, puppetdb to nftables [[phab:T427799|T427799]]
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-01 ===
* 20:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-05-27 ===
* 20:33 andrewbogott: rebuilt pywikibot for v11.3.0  [[phab:T426547|T426547]]
* 12:12 godog: begin roll-restart webservices with new default requests - [[phab:T420565|T420565]]
* 12:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 12:02 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 09:11 godog: delete obsolete limitranges - [[phab:T427352|T427352]]

=== 2026-05-21 ===
* 12:46 taavi: renew prometheus certificate
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3
* 10:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:48 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 00:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api

=== 2026-05-20 ===
* 23:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 00:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-05-19 ===
* 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-05-18 ===
* 10:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]])
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]])

=== 2026-05-16 ===
* 00:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers

=== 2026-05-15 ===
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 19:02 taavi: rebooting bastions and k8s workers to pick up kernel updates

=== 2026-05-14 ===
* 16:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 13:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-05-13 ===
* 12:07 godog: resume restarting webservices using default memory requests - [[phab:T420565|T420565]]
* 08:46 godog: restart sample webservices with new memory requests https://phabricator.wikimedia.org/P92497 - [[phab:T420565|T420565]]
* 08:36 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 08:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2026-05-12 ===
* 23:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 22:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 22:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2026-05-11 ===
* 00:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config
* 00:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 00:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config

=== 2026-05-07 ===
* 12:02 taavi: draining tools-k8s-worker-106 to investigate [[phab:T425172|T425172]]

=== 2026-05-05 ===
* 04:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 04:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 02:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 02:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 02:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2026-04-28 ===
* 10:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-04-23 ===
* 15:59 andrewbogott: hard rebooting tools-puppetserver-01.tools, it seems to have crashed
* 09:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99)
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:12 taavi: uninstall ingress-nginx-gen2 from the cluster [[phab:T392356|T392356]]
* 08:08 taavi: delete all ingress objects [[phab:T392356|T392356]]

=== 2026-04-21 ===
* 14:06 taavi: save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json [[phab:T392356|T392356]]
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli

=== 2026-04-20 ===
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 15:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2026-04-16 ===
* 13:09 taavi: bump istio traffic percentage 75% -&gt; 100% [[phab:T392356|T392356]]

=== 2026-04-15 ===
* 10:45 taavi: bump istio traffic percentage 50% -&gt; 75% [[phab:T392356|T392356]]

=== 2026-04-13 ===
* 09:11 taavi: bump istio traffic percentage 25% -&gt; 50% [[phab:T392356|T392356]]
* 07:33 taavi: bump istio traffic percentage 10% -&gt; 25% [[phab:T392356|T392356]]

=== 2026-04-10 ===
* 14:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-04-09 ===
* 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2026-04-08 ===
* 17:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api

=== 2026-04-07 ===
* 23:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:59 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0)
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 15:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 15:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]])
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrewbogott: replacing etcd nodes with bookworm-based VMs
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 07:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 07:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-04-02 ===
* 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 16:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-04-01 ===
* 18:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2026-03-31 ===
* 18:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2026-03-30 ===
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 dcaro: removing wal from prometheus nodes to restart them

=== 2026-03-26 ===
* 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2026-03-25 ===
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04
* 13:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04

=== 2026-03-24 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-03-23 ===
* 11:16 taavi: send 10% of traffic to istio [[phab:T392356|T392356]]
* 10:53 taavi: send 5% of traffic to istio [[phab:T392356|T392356]]
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-03-19 ===
* 20:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
* 09:07 taavi: fixing 2 tools still running ruby2.1 image to use that instead of 'ruby2' in service.manifest
* 08:52 taavi: fixing 2 tools still running ruby2.5 image to use that instead of 'ruby25' in service.manifest
* 08:49 taavi: fixing 12 tools still running node6 image to use that instead of 'nodejs' in service.manifest
* 08:38 taavi: fixing 12 tools still running golang1.11 image to use that instead of 'golang111' in service.manifest
* 08:36 taavi: fixing 60 tools still running python3.4 image to use 'python3.4' instead of 'python' in service.manifest

=== 2026-03-18 ===
* 12:00 taavi: restarting existing web services to backfill HTTPRoute resources [[phab:T392356|T392356]]
* 07:37 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:37 filippo@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-83.tools.eqiad1.wikimedia.cloud to the cluster
* 07:23 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T419824|T419824]])

=== 2026-03-17 ===
* 12:43 taavi: shutdown tools-package-builder-04 [[phab:T401819|T401819]]

=== 2026-03-15 ===
* 03:10 andrewbogott: rebooting tools-redis-6, VM is in state ERROR

=== 2026-03-13 ===
* 22:04 taavi: reboot tools-bastion-15 [[phab:T420044|T420044]]
* 19:06 taavi: reboot tools-bastion-14 [[phab:T420044|T420044]]

=== 2026-03-12 ===
* 13:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 13:50 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2026-03-10 ===
* 11:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli
* 09:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-03-09 ===
* 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster
* 13:30 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster
* 13:19 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster
* 13:07 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster

=== 2026-03-06 ===
* 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system

=== 2026-03-05 ===
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api

=== 2026-03-04 ===
* 20:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:44 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:29 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:14 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:13 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:12 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=97)
* 19:11 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:08 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 18:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 18:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 18:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 18:17 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 17:38 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 14:54 dcaro: increase object quota to 400k ([[phab:T418528|T418528]])
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api

=== 2026-03-03 ===
* 20:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 20:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-03-02 ===
* 17:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-02-26 ===
* 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api

=== 2026-02-25 ===
* 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0
* 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0
* 14:09 taavi: taavi@tools-imagebuilder-2:~$ sudo docker system prune -a # reclaiming disk space
* 14:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0
* 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0

=== 2026-02-24 ===
* 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-02-20 ===
* 20:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 20:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 20:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 19:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}

=== 2026-02-11 ===
* 21:49 taavi: remove hiera override still allowing ssh agent forwarding onto toolforge bastions [[phab:T198138|T198138]]

=== 2026-02-05 ===
* 19:08 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 18:48 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
* 16:42 volans: re-enabling puppet on NFS workers to update the infra-tracing-nfs

=== 2026-02-04 ===
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3
* 14:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3

=== 2026-02-03 ===
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7

=== 2026-01-28 ===
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2026-01-23 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2026-01-22 ===
* 18:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 18:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2026-01-15 ===
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-01-14 ===
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]])
* 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]])
* 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 15:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 15:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 15:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]])
* 14:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]])
* 14:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]])
* 13:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]])

=== 2026-01-12 ===
* 17:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 17:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 17:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 16:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 16:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2026-01-06 ===
* 15:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 14:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 13:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 13:48 andrewbogott: removing tools-k8s-etcd-24 in prep for rebuilding cloudvirtlocal1003
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 03:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 03:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 02:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 02:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 01:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 01:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 01:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])

=== 2026-01-05 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 23:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 23:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 23:01 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97)
* 22:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])

=== 2025-12-18 ===
* 11:13 godog: bump max objects quota to 200k
* 11:05 godog: bump object quota to 500G

=== 2025-12-17 ===
* 17:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]])
* 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.12.1 ([[phab:T399313|T399313]])
* 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.6.3 ([[phab:T399313|T399313]])
* 17:53 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]])

=== 2025-12-15 ===
* 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T412695|T412695]])
* 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.17.0 ([[phab:T412695|T412695]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T412695|T412695]])

=== 2025-12-14 ===
* 02:14 andrewbogott: running 'kubectl rollout restart -n envvars-admission deployment/envvars-admission' in response to an envvars alert

=== 2025-12-11 ===
* 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-12-04 ===
* 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 20:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 19:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])

=== 2025-12-03 ===
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]])
* 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]])

=== 2025-12-02 ===
* 20:22 andrewbogott: stop/starting harbordb1 to fix presumed mtu mismatch
* 20:06 andrewbogott: rebooting tools-harbordb1 to aid with host draining
* 08:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.11.3 ([[phab:T399313|T399313]])
* 08:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.11.3 ([[phab:T399313|T399313]])
* 08:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.11.3 ([[phab:T399313|T399313]])

=== 2025-12-01 ===
* 22:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 ([[phab:T399313|T399313]])
* 22:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 ([[phab:T399313|T399313]])
* 22:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 ([[phab:T399313|T399313]])
* 16:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 16:26 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing

=== 2025-11-27 ===
* 12:15 volans: [continue] on the haproxy nodes
* 12:15 volans: temporarily disabling puppet to deploy gerrit {{Gerrit|1211610}}

=== 2025-11-26 ===
* 14:48 volans: enabled infra-tracing-nfs on all nfs workers after testing it on few hosts
* 09:46 dhinus: restarting tools-db-6 to apply a config change [[phab:T409922|T409922]]

=== 2025-11-25 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 02:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 02:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 01:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 01:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2025-11-24 ===
* 10:24 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 10:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing

=== 2025-11-20 ===
* 18:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34
* 17:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 16:55 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
* 16:45 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 16:36 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:56 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:51 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:37 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:37 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 15:17 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:01 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 14:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 14:46 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 14:40 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-11-19 ===
* 16:19 andrewbogott: increased object count quota to 100,000
* 16:03 andrewbogott: increased object storage quota to 200GB
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-11-18 ===
* 18:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-harbor-2 (cluster eqiad1)
* 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-harbor-2 (cluster eqiad1)
* 14:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-9 (cluster eqiad1)
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-9 (cluster eqiad1)
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-113 (cluster eqiad1)
* 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-113 (cluster eqiad1)
* 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-113
* 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-113
* 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-112 (cluster eqiad1)
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-112 (cluster eqiad1)
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-112
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-112
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-82 (cluster eqiad1)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-82 (cluster eqiad1)
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-82
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-82
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1)
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-81 (cluster eqiad1)
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-81
* 14:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-81
* 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-80 (cluster eqiad1)
* 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-80 (cluster eqiad1)
* 14:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-80
* 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-80
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-8 (cluster eqiad1)
* 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-8 (cluster eqiad1)
* 12:05 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:56 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:24 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 11:19 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-legacy-redirector-3 (cluster eqiad1)
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legacy-redirector-3 (cluster eqiad1)
* 10:11 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.instance.stop_start (exit_code=97) vm tools-legaci-redirector-3 (cluster eqiad1)
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legaci-redirector-3 (cluster eqiad1)

=== 2025-11-17 ===
* 18:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 18:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T409981|T409981]])
* 10:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T409981|T409981]])

=== 2025-11-14 ===
* 16:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-4 ([[phab:T409287|T409287]])
* 16:26 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-4 ([[phab:T409287|T409287]])

=== 2025-11-13 ===
* 15:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1)
* 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1)
* 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-bastion-14 (cluster eqiad1)
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-bastion-14 (cluster eqiad1)
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-7 (cluster eqiad1)
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-7 (cluster eqiad1)
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-8 (cluster eqiad1)
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-8 (cluster eqiad1)

=== 2025-11-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-11-11 ===
* 15:28 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 ([[phab:T399313|T399313]])
* 15:28 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 ([[phab:T399313|T399313]])
* 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 ([[phab:T399313|T399313]])

=== 2025-11-10 ===
* 22:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 22:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99)
* 22:16 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest
* 22:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 22:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99)
* 22:14 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest
* 22:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 22:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99)
* 22:08 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest:latest
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 21:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 21:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 21:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 20:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-11-07 ===
* 11:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 11:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]])
* 11:35 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]])
* 11:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 ([[phab:T409287|T409287]])
* 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 ([[phab:T409287|T409287]])

=== 2025-11-06 ===
* 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-11-05 ===
* 19:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:40 taavi: taavi@tools-bastion-15:~ $ sudo loginctl terminate-user damian
* 14:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 14:53 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]])

=== 2025-11-04 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 17:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 03:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
* 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
* 01:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
* 00:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 00:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 00:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 00:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-11-03 ===
* 22:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 22:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 22:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 22:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 21:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 21:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 18:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 11:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 11:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-10-30 ===
* 18:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 11:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 11:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2025-10-29 ===
* 18:39 taavi: kick off script to rebuild all pre-built images, including [[phab:T407707|T407707]]
* 16:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]])
* 16:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]])
* 16:27 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T408669|T408669]])
* 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6
* 12:48 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6
* 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6
* 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6
* 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6
* 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6
* 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 12:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2025-10-28 ===
* 19:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:07 taavi: delete paws, paws-master security groups, long obsolete as paws is now in a separae project
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-10-27 ===
* 22:09 taavi: copy toolviews database hiera data to a place where haproxy nodes can see them [[phab:T408454|T408454]]
* 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:16 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]])

=== 2025-10-24 ===
* 20:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t
* 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27
* 17:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to
* 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106
* 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103
* 13:37 andrewbogott: rebooting clouddumps100[12] for [[phab:T407110|T407110]]

=== 2025-10-23 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:11 taavi: deleting old nginx front proxy instances [[phab:T283948|T283948]]
* 10:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-10-22 ===
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 15:56 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3
* 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:35 taavi: moving toolforge traffic to haproxy directly [[phab:T283948|T283948]]
* 07:00 godog: delete tools-nfs-2 - [[phab:T404584|T404584]]

=== 2025-10-21 ===
* 08:53 godog: shut down tools-nfs-2 - [[phab:T404584|T404584]]
* 07:52 godog: tools-nfs-3 is back - [[phab:T404584|T404584]]
* 07:49 godog: resize tools-nfs-3 to match tools-nfs-2 (g4.cores16.ram64.disk20.10xiops) - [[phab:T404584|T404584]]
* 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-10-20 ===
* 16:31 taavi: make logrotate run hourly on haproxy nodes [[phab:T284558|T284558]]

=== 2025-10-16 ===
* 12:01 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-10-15 ===
* 08:03 godog: tools-nfs-3 is back - [[phab:T404584|T404584]]
* 08:00 godog: resize tools-nfs-3 - [[phab:T404584|T404584]]

=== 2025-10-14 ===
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:45 godog: update nfs-tools.wmcloud.org and nfs.svc.toolforge.org proxied to point to tools-nfs-3

=== 2025-10-13 ===
* 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:17 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, too
* 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools)
* 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools)
* 09:13 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:13 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools)
* 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools)
* 09:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools)
* 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools)
* 08:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools)
* 08:59 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools)
* 08:58 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:58 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:57 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:10 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]])
* 08:05 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]])
* 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:03 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]])
* 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:01 godog: switch NFS from tools-nfs-2 to tools-nfs-3 - [[phab:T404584|T404584]]
* 07:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81

=== 2025-10-10 ===
* 09:22 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1
* 09:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1

=== 2025-10-09 ===
* 08:21 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 08:15 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2025-10-08 ===
* 21:47 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76
* 21:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76
* 12:27 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]]
* 10:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 06:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71
* 06:43 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71

=== 2025-10-07 ===
* 18:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11
* 15:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69
* 15:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69
* 14:51 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69
* 14:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:58 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 08:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-10-06 ===
* 12:06 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7
* 11:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7
* 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 08:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 08:18 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76
* 07:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76

=== 2025-10-03 ===
* 12:51 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-6
* 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-6
* 12:45 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-5
* 09:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-5

=== 2025-10-02 ===
* 13:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-haproxy-7
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-haproxy-7
* 13:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 09:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55
* 08:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55

=== 2025-10-01 ===
* 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-09-30 ===
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67
* 08:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67

=== 2025-09-29 ===
* 13:23 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 13:23 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 11:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:34 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 07:00 godog: kick stuck nfs workers from clouddumps1001

=== 2025-09-28 ===
* 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:13 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]])
* 08:10 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]])
* 08:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:07 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 08:07 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-09-25 ===
* 18:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 15:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 11:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-09-24 ===
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 20:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 17:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 17:30 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43
* 17:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 17:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43
* 17:07 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]])
* 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]])
* 13:49 dcaro: patched all tools with new resource defaults, everything looks good
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:09 dcaro: depolyed jobs-api change to default resources, patching existing jobs
* 13:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 12:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 03:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12

=== 2025-09-23 ===
* 20:08 andrewbogott: creating puppetdbpostgres and adding it to tools-puppetdb-2 to store postgres data; the root volume of that VM was filling up and causing widespread puppet issues
* 01:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-09-22 ===
* 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-09-21 ===
* 09:17 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2
* 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2
* 03:16 dcaro: acking and silencing CPU capacity alerts to handle on Monday, they should not page
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 01:46 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-113.tools.eqiad1.wikimedia.cloud to the cluster
* 01:36 andrewbogott: adding additional worker node in response to repeated capacity alerts
* 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2025-09-19 ===
* 13:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 13:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11

=== 2025-09-18 ===
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools)
* 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools)
* 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 09:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 08:52 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3
* 08:34 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3
* 06:47 taavi: delete tools-sgebastion-10 [[phab:T314665|T314665]]

=== 2025-09-17 ===
* 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32
* 12:53 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32
* 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools)
* 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools)
* 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10
* 08:08 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10

=== 2025-09-16 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:57 taavi: delete tools-sgebastion puppet prefix [[phab:T314665|T314665]]
* 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 taavi: shutdown tools-sgebastion-10 [[phab:T314665|T314665]]
* 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-13
* 14:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-13
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-12
* 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-12
* 07:11 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75
* 06:57 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75
* 06:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 06:51 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-09-15 ===
* 16:22 taavi: reboot old bastions to kick long-living connections into newer ones
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 12:47 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66
* 12:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66

=== 2025-09-12 ===
* 08:49 taavi: pointing login.toolforge.org to tools-bastion-15 [[phab:T392510|T392510]]
* 08:33 taavi: pointing dev.toolforge.org to tools-bastion-14 [[phab:T392510|T392510]]
* 07:14 godog: uncordon tools-k8s-worker-nfs-53 after failed cookbook (?) yesterday

=== 2025-09-11 ===
* 14:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:36 godog: drain/reboot tools-k8s-worker-nfs-46 - [[phab:T404322|T404322]]
* 14:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:22 andrewbogott: actually I didn't drain tools-k8s-worker-nfs-53 because the alert cleared on its own
* 14:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-53
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53
* 14:21 andrewbogott: draining/rebooting tools-k8s-worker-nfs-53 because of procs in D state
* 13:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53
* 13:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-14.tools.eqiad1.wikimedia.cloud
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-14.tools.eqiad1.wikimedia.cloud
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-15.tools.eqiad1.wikimedia.cloud
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-15.tools.eqiad1.wikimedia.cloud
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-09-10 ===
* 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T403964|T403964]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:45 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:31 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]])

=== 2025-09-09 ===
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:55 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) ([[phab:T404047|T404047]])
* 08:55 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot ([[phab:T404047|T404047]])

=== 2025-09-08 ===
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 15:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:16 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97)
* 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 12:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]])
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]])
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]])
* 11:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]])
* 11:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 10:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 10:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]])
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]])
* 09:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]])
* 09:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]])
* 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:40 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 09:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 08:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 08:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 08:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97)
* 08:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests

=== 2025-09-06 ===
* 23:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35
* 23:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35

=== 2025-09-05 ===
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-09-04 ===
* 19:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:52 dcaro: added 'disable-ssl' to tools replica.my.cnf

=== 2025-09-03 ===
* 17:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 09:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-09-02 ===
* 17:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 17:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-29 ===
* 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting

=== 2025-08-28 ===
* 16:52 taavi: rebuild tcl, mariadb images on top of trixie [[phab:T400256|T400256]]
* 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-08-27 ===
* 18:05 taavi: copy missing aptly packages to trixie-&lt;nowiki&gt;{&lt;/nowiki&gt;tools,toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;
* 11:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-08-26 ===
* 13:42 dcaro: extended object storage quota to 100G ([[phab:T402923|T402923]])
* 10:25 dhinus: shut down tools-harbor-1 (no longer used)

=== 2025-08-25 ===
* 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-81
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-81

=== 2025-08-21 ===
* 12:28 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:31 godog: reboot nfs workers to reset processes stuck in D state
* 07:28 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 04:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80
* 03:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80

=== 2025-08-20 ===
* 08:09 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]])

=== 2025-08-19 ===
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 15:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 15:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:50 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:37 dcaro: flipped the tools-harbor.wmcloud.org endpoint to point to tools-harbor-2 ([[phab:T350687|T350687]])
* 14:22 Raymond_Ndibe: setting tools-harbor-1 as read-only ([[phab:T350687|T350687]])
* 13:24 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 13:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-18 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]])
* 17:49 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.16.0 ([[phab:T362869|T362869]])
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]])
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]])
* 17:48 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/metrics-server:v0.7.2 ([[phab:T362869|T362869]])
* 17:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]])
* 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]])
* 16:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]])
* 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-harbor-2.tools.eqiad1.wikimedia.cloud
* 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud

=== 2025-08-16 ===
* 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-111
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-111
* 21:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-111
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-111

=== 2025-08-15 ===
* 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 19:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103

=== 2025-08-14 ===
* 15:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 15:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41
* 11:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41
* 11:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 11:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 02:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-13 ===
* 16:36 dcaro: reverting jobs-api release ([[phab:T401846|T401846]])
* 11:18 taavi: delete tools-prometheus-6, shutdown for a while
* 08:51 godog: bounce stashbot
* 08:33 godog: refresh machine-id on tools-k8s-worker-[102-103,105-112].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-3,5,7-14,16-17,19,21-24,26-27,32-48,50,53-55 ,57-58,61,65-82].tools.eqiad1.wikimedia.cloud

=== 2025-08-12 ===
* 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 15:34 taavi: building initial trixie based images [[phab:T400255|T400255]]
* 12:50 dcaro: redepoly kyverno ([[phab:T394787|T394787]])
* 12:49 dcaro: manually migrate cleanuppolicies.kyverno.io and clustercleanuppolicies.kyverno.io (using kyverno cli) ([[phab:T394787|T394787]])
* 10:01 dcaro: starting upgrade for kyverno ([[phab:T394787|T394787]])
* 10:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli

=== 2025-08-11 ===
* 12:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud
* 08:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud
* 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58
* 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58

=== 2025-08-08 ===
* 06:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-07 ===
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67
* 14:20 andrewbogott: draining and rebooting tools-k8s-worker-nfs-67
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-06 ===
* 17:54 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 17:53 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 17:41 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]])
* 17:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]])
* 17:39 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]])
* 17:38 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]])
* 17:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-05 ===
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24
* 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24
* 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 03:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 02:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 02:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 02:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 02:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 02:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 02:30 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 02:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 01:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 01:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-08-04 ===
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]])
* 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]])
* 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2
* 11:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2

=== 2025-08-01 ===
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli

=== 2025-07-31 ===
* 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 16:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 04:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 04:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 04:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 04:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 04:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 04:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 04:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 04:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 04:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-07-30 ===
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-29 ===
* 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 15:56 andrewbogott: draining and restarting  tools-k8s-worker-nfs-74
* 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32
* 15:32 andrewbogott: draining and restarting  tools-k8s-worker-nfs-58 and  tools-k8s-worker-nfs-32
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:16 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-82.tools.eqiad1.wikimedia.cloud to the cluster
* 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:06 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:05 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-81.tools.eqiad1.wikimedia.cloud to the cluster
* 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.quota_increase
* 12:39 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:31 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-80.tools.eqiad1.wikimedia.cloud to the cluster
* 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:00 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:28 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:07 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:59 wmbot~dcaro@acme: Added a new k8s worker tools-k8s-worker-112.tools.eqiad1.wikimedia.cloud to the cluster
* 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 08:49 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:14 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2025-07-28 ===
* 20:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 19:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 11:58 taavi: update pywikibot image to 10.2.0 [[phab:T396933|T396933]]

=== 2025-07-26 ===
* 07:16 wmbot~root@toolforge: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:16 wmbot~root@toolforge: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli

=== 2025-07-23 ===
* 18:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-07-21 ===
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-07-19 ===
* 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 13:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-07-18 ===
* 10:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44

=== 2025-07-14 ===
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78

=== 2025-07-13 ===
* 03:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12

=== 2025-07-11 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37
* 09:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37

=== 2025-07-09 ===
* 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli
* 10:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli
* 09:55 dcaro: adding arch arm64 to all toolforge repos ([[phab:T398016|T398016]])
* 09:40 dcaro: added arch arm64 to jessie-tools repo ([[phab:T398016|T398016]])

=== 2025-07-08 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-07 ===
* 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53
* 17:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53
* 16:39 dcaro: pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest
* 16:05 dcaro: clearing images from tools-imagebuilder-2 as it's out of space
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>r0w1gx41v8hto8uihycvcb1wvald549</sha1>
    </revision>
    <revision>
      <id>2435524</id>
      <parentid>2435495</parentid>
      <timestamp>2026-07-12T06:28:39Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>dcaro: restart haproxy on haproxy-8</comment>
      <origin>2435524</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="787286" sha1="6s00g7ef4spgowcpqd1c1yktlk7nwwp" xml:space="preserve">=== 2026-07-12 ===
* 06:28 dcaro: restart haproxy on haproxy-8

=== 2026-07-11 ===
* 17:34 dcaro: rebooting haproxy-8

=== 2026-07-09 ===
* 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 12:53 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:27 taavi: rebooting bastions for kernel upgrades
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-08 ===
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-07 ===
* 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-06 ===
* 17:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-07-02 ===
* 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])
* 10:18 taavi: deploy latest striker with [[phab:T430120|T430120]] [[phab:T409341|T409341]] fixes
* 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-30 ===
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-29 ===
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-25 ===
* 17:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 08:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]])
* 08:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]])

=== 2026-06-24 ===
* 16:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2026-06-23 ===
* 08:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-22 ===
* 10:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-18 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-15 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 12:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]])
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]])
* 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:45 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]])
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]])
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]])
* 11:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]])
* 09:53 godog: delete tools-checker puppet prefix - [[phab:T313030|T313030]]
* 09:52 godog: delete tools-checker-5 - [[phab:T313030|T313030]]

=== 2026-06-11 ===
* 13:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]])
* 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]])

=== 2026-06-09 ===
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 11:01 taavi: starting Kyverno upgrade [[phab:T427928|T427928]]

=== 2026-06-04 ===
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway
* 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 12:15 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway
* 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:06 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7
* 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4
* 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4
* 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4
* 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4
* 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4
* 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4
* 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:02 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4
* 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry

=== 2026-06-02 ===
* 15:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 10:33 taavi: migrate acme-chief, puppetdb to nftables [[phab:T427799|T427799]]
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-06-01 ===
* 20:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-05-27 ===
* 20:33 andrewbogott: rebuilt pywikibot for v11.3.0  [[phab:T426547|T426547]]
* 12:12 godog: begin roll-restart webservices with new default requests - [[phab:T420565|T420565]]
* 12:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 12:02 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 09:11 godog: delete obsolete limitranges - [[phab:T427352|T427352]]

=== 2026-05-21 ===
* 12:46 taavi: renew prometheus certificate
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3
* 10:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:48 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 00:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api

=== 2026-05-20 ===
* 23:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 00:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-05-19 ===
* 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-05-18 ===
* 10:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]])
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]])

=== 2026-05-16 ===
* 00:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers

=== 2026-05-15 ===
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 19:02 taavi: rebooting bastions and k8s workers to pick up kernel updates

=== 2026-05-14 ===
* 16:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 13:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-05-13 ===
* 12:07 godog: resume restarting webservices using default memory requests - [[phab:T420565|T420565]]
* 08:46 godog: restart sample webservices with new memory requests https://phabricator.wikimedia.org/P92497 - [[phab:T420565|T420565]]
* 08:36 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 08:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2026-05-12 ===
* 23:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 22:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 22:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2026-05-11 ===
* 00:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config
* 00:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 00:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config

=== 2026-05-07 ===
* 12:02 taavi: draining tools-k8s-worker-106 to investigate [[phab:T425172|T425172]]

=== 2026-05-05 ===
* 04:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 04:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 02:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 02:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 02:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2026-04-28 ===
* 10:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-04-23 ===
* 15:59 andrewbogott: hard rebooting tools-puppetserver-01.tools, it seems to have crashed
* 09:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99)
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:12 taavi: uninstall ingress-nginx-gen2 from the cluster [[phab:T392356|T392356]]
* 08:08 taavi: delete all ingress objects [[phab:T392356|T392356]]

=== 2026-04-21 ===
* 14:06 taavi: save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json [[phab:T392356|T392356]]
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli

=== 2026-04-20 ===
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 15:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2026-04-16 ===
* 13:09 taavi: bump istio traffic percentage 75% -&gt; 100% [[phab:T392356|T392356]]

=== 2026-04-15 ===
* 10:45 taavi: bump istio traffic percentage 50% -&gt; 75% [[phab:T392356|T392356]]

=== 2026-04-13 ===
* 09:11 taavi: bump istio traffic percentage 25% -&gt; 50% [[phab:T392356|T392356]]
* 07:33 taavi: bump istio traffic percentage 10% -&gt; 25% [[phab:T392356|T392356]]

=== 2026-04-10 ===
* 14:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-04-09 ===
* 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2026-04-08 ===
* 17:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api

=== 2026-04-07 ===
* 23:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:59 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]])
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0)
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 15:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 15:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]])
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrewbogott: replacing etcd nodes with bookworm-based VMs
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 07:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 07:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-04-02 ===
* 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 16:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-04-01 ===
* 18:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2026-03-31 ===
* 18:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2026-03-30 ===
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 dcaro: removing wal from prometheus nodes to restart them

=== 2026-03-26 ===
* 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2
* 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2026-03-25 ===
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04
* 13:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04

=== 2026-03-24 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-03-23 ===
* 11:16 taavi: send 10% of traffic to istio [[phab:T392356|T392356]]
* 10:53 taavi: send 5% of traffic to istio [[phab:T392356|T392356]]
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway

=== 2026-03-19 ===
* 20:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
* 09:07 taavi: fixing 2 tools still running ruby2.1 image to use that instead of 'ruby2' in service.manifest
* 08:52 taavi: fixing 2 tools still running ruby2.5 image to use that instead of 'ruby25' in service.manifest
* 08:49 taavi: fixing 12 tools still running node6 image to use that instead of 'nodejs' in service.manifest
* 08:38 taavi: fixing 12 tools still running golang1.11 image to use that instead of 'golang111' in service.manifest
* 08:36 taavi: fixing 60 tools still running python3.4 image to use 'python3.4' instead of 'python' in service.manifest

=== 2026-03-18 ===
* 12:00 taavi: restarting existing web services to backfill HTTPRoute resources [[phab:T392356|T392356]]
* 07:37 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:37 filippo@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-83.tools.eqiad1.wikimedia.cloud to the cluster
* 07:23 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T419824|T419824]])

=== 2026-03-17 ===
* 12:43 taavi: shutdown tools-package-builder-04 [[phab:T401819|T401819]]

=== 2026-03-15 ===
* 03:10 andrewbogott: rebooting tools-redis-6, VM is in state ERROR

=== 2026-03-13 ===
* 22:04 taavi: reboot tools-bastion-15 [[phab:T420044|T420044]]
* 19:06 taavi: reboot tools-bastion-14 [[phab:T420044|T420044]]

=== 2026-03-12 ===
* 13:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 13:50 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2026-03-10 ===
* 11:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli
* 09:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-03-09 ===
* 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster
* 13:30 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster
* 13:19 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster
* 13:07 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster

=== 2026-03-06 ===
* 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system

=== 2026-03-05 ===
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api

=== 2026-03-04 ===
* 20:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:44 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:29 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:14 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:13 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:12 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=97)
* 19:11 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:08 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 19:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 19:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 18:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 18:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 18:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster
* 18:17 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 17:38 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 14:54 dcaro: increase object quota to 400k ([[phab:T418528|T418528]])
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api

=== 2026-03-03 ===
* 20:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 20:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-03-02 ===
* 17:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-02-26 ===
* 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api

=== 2026-02-25 ===
* 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0
* 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0
* 14:09 taavi: taavi@tools-imagebuilder-2:~$ sudo docker system prune -a # reclaiming disk space
* 14:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0
* 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0

=== 2026-02-24 ===
* 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-02-20 ===
* 20:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 20:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 20:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24
* 19:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}
* 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}}

=== 2026-02-11 ===
* 21:49 taavi: remove hiera override still allowing ssh agent forwarding onto toolforge bastions [[phab:T198138|T198138]]

=== 2026-02-05 ===
* 19:08 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 18:48 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
* 16:42 volans: re-enabling puppet on NFS workers to update the infra-tracing-nfs

=== 2026-02-04 ===
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3
* 14:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3

=== 2026-02-03 ===
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7

=== 2026-01-28 ===
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2026-01-23 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2026-01-22 ===
* 18:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 18:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2026-01-15 ===
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2026-01-14 ===
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]])
* 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]])
* 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 15:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 15:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 15:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]])
* 14:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]])
* 14:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]])
* 14:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]])
* 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]])
* 13:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]])

=== 2026-01-12 ===
* 17:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 17:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 17:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 16:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 16:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2026-01-06 ===
* 15:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 14:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 13:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 13:48 andrewbogott: removing tools-k8s-etcd-24 in prep for rebuilding cloudvirtlocal1003
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 03:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 03:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 02:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 02:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 01:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 01:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])
* 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 01:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]])

=== 2026-01-05 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 23:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 23:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 23:01 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97)
* 22:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])

=== 2025-12-18 ===
* 11:13 godog: bump max objects quota to 200k
* 11:05 godog: bump object quota to 500G

=== 2025-12-17 ===
* 17:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]])
* 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.12.1 ([[phab:T399313|T399313]])
* 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.6.3 ([[phab:T399313|T399313]])
* 17:53 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]])

=== 2025-12-15 ===
* 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T412695|T412695]])
* 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.17.0 ([[phab:T412695|T412695]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T412695|T412695]])

=== 2025-12-14 ===
* 02:14 andrewbogott: running 'kubectl rollout restart -n envvars-admission deployment/envvars-admission' in response to an envvars alert

=== 2025-12-11 ===
* 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-12-04 ===
* 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 20:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 19:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]])

=== 2025-12-03 ===
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0)
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]])
* 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]])

=== 2025-12-02 ===
* 20:22 andrewbogott: stop/starting harbordb1 to fix presumed mtu mismatch
* 20:06 andrewbogott: rebooting tools-harbordb1 to aid with host draining
* 08:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.11.3 ([[phab:T399313|T399313]])
* 08:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.11.3 ([[phab:T399313|T399313]])
* 08:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.11.3 ([[phab:T399313|T399313]])

=== 2025-12-01 ===
* 22:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 ([[phab:T399313|T399313]])
* 22:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 ([[phab:T399313|T399313]])
* 22:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 ([[phab:T399313|T399313]])
* 16:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 16:26 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing

=== 2025-11-27 ===
* 12:15 volans: [continue] on the haproxy nodes
* 12:15 volans: temporarily disabling puppet to deploy gerrit {{Gerrit|1211610}}

=== 2025-11-26 ===
* 14:48 volans: enabled infra-tracing-nfs on all nfs workers after testing it on few hosts
* 09:46 dhinus: restarting tools-db-6 to apply a config change [[phab:T409922|T409922]]

=== 2025-11-25 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 02:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 02:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 01:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 01:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2025-11-24 ===
* 10:24 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 10:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing

=== 2025-11-20 ===
* 18:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34
* 17:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
* 16:55 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
* 16:45 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 16:36 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:56 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:51 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:37 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:37 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 15:17 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:01 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 14:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 14:46 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 14:40 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-11-19 ===
* 16:19 andrewbogott: increased object count quota to 100,000
* 16:03 andrewbogott: increased object storage quota to 200GB
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-11-18 ===
* 18:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-harbor-2 (cluster eqiad1)
* 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-harbor-2 (cluster eqiad1)
* 14:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-9 (cluster eqiad1)
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-9 (cluster eqiad1)
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-113 (cluster eqiad1)
* 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-113 (cluster eqiad1)
* 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-113
* 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-113
* 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-112 (cluster eqiad1)
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-112 (cluster eqiad1)
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-112
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-112
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-82 (cluster eqiad1)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-82 (cluster eqiad1)
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-82
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-82
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1)
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-81 (cluster eqiad1)
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-81
* 14:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-81
* 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-80 (cluster eqiad1)
* 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-80 (cluster eqiad1)
* 14:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-80
* 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-80
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-8 (cluster eqiad1)
* 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-8 (cluster eqiad1)
* 12:05 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:56 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:24 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 11:19 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-legacy-redirector-3 (cluster eqiad1)
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legacy-redirector-3 (cluster eqiad1)
* 10:11 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.instance.stop_start (exit_code=97) vm tools-legaci-redirector-3 (cluster eqiad1)
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legaci-redirector-3 (cluster eqiad1)

=== 2025-11-17 ===
* 18:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 18:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T409981|T409981]])
* 10:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T409981|T409981]])

=== 2025-11-14 ===
* 16:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-4 ([[phab:T409287|T409287]])
* 16:26 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-4 ([[phab:T409287|T409287]])

=== 2025-11-13 ===
* 15:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1)
* 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1)
* 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-bastion-14 (cluster eqiad1)
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-bastion-14 (cluster eqiad1)
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-7 (cluster eqiad1)
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-7 (cluster eqiad1)
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-8 (cluster eqiad1)
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-8 (cluster eqiad1)

=== 2025-11-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-11-11 ===
* 15:28 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 ([[phab:T399313|T399313]])
* 15:28 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 ([[phab:T399313|T399313]])
* 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 ([[phab:T399313|T399313]])

=== 2025-11-10 ===
* 22:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 22:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99)
* 22:16 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest
* 22:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 22:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99)
* 22:14 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest
* 22:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 22:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99)
* 22:08 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest:latest
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 21:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 21:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 21:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 20:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-11-07 ===
* 11:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 11:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]])
* 11:35 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]])
* 11:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 ([[phab:T409287|T409287]])
* 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 ([[phab:T409287|T409287]])

=== 2025-11-06 ===
* 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-11-05 ===
* 19:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:40 taavi: taavi@tools-bastion-15:~ $ sudo loginctl terminate-user damian
* 14:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 14:53 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]])
* 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]])

=== 2025-11-04 ===
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 17:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 03:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
* 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
* 01:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
* 00:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 00:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 00:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 00:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-11-03 ===
* 22:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 22:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 22:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 22:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 21:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 21:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 18:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 11:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 11:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-10-30 ===
* 18:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api
* 11:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api
* 11:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api

=== 2025-10-29 ===
* 18:39 taavi: kick off script to rebuild all pre-built images, including [[phab:T407707|T407707]]
* 16:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]])
* 16:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]])
* 16:27 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T408669|T408669]])
* 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6
* 12:48 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6
* 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6
* 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6
* 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6
* 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6
* 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 12:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2025-10-28 ===
* 19:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:07 taavi: delete paws, paws-master security groups, long obsolete as paws is now in a separae project
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-10-27 ===
* 22:09 taavi: copy toolviews database hiera data to a place where haproxy nodes can see them [[phab:T408454|T408454]]
* 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:16 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]])

=== 2025-10-24 ===
* 20:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t
* 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27
* 17:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to
* 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106
* 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103
* 13:37 andrewbogott: rebooting clouddumps100[12] for [[phab:T407110|T407110]]

=== 2025-10-23 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:11 taavi: deleting old nginx front proxy instances [[phab:T283948|T283948]]
* 10:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-10-22 ===
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 15:56 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3
* 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:35 taavi: moving toolforge traffic to haproxy directly [[phab:T283948|T283948]]
* 07:00 godog: delete tools-nfs-2 - [[phab:T404584|T404584]]

=== 2025-10-21 ===
* 08:53 godog: shut down tools-nfs-2 - [[phab:T404584|T404584]]
* 07:52 godog: tools-nfs-3 is back - [[phab:T404584|T404584]]
* 07:49 godog: resize tools-nfs-3 to match tools-nfs-2 (g4.cores16.ram64.disk20.10xiops) - [[phab:T404584|T404584]]
* 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-10-20 ===
* 16:31 taavi: make logrotate run hourly on haproxy nodes [[phab:T284558|T284558]]

=== 2025-10-16 ===
* 12:01 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-10-15 ===
* 08:03 godog: tools-nfs-3 is back - [[phab:T404584|T404584]]
* 08:00 godog: resize tools-nfs-3 - [[phab:T404584|T404584]]

=== 2025-10-14 ===
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:45 godog: update nfs-tools.wmcloud.org and nfs.svc.toolforge.org proxied to point to tools-nfs-3

=== 2025-10-13 ===
* 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:17 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, too
* 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools)
* 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools)
* 09:13 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:13 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools)
* 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools)
* 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools)
* 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools)
* 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools)
* 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools)
* 09:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools)
* 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools)
* 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools)
* 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools)
* 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools)
* 08:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools)
* 08:59 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools)
* 08:58 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:58 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:57 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools)
* 08:10 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]])
* 08:05 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]])
* 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:03 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]])
* 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]])
* 08:01 godog: switch NFS from tools-nfs-2 to tools-nfs-3 - [[phab:T404584|T404584]]
* 07:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81

=== 2025-10-10 ===
* 09:22 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1
* 09:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1

=== 2025-10-09 ===
* 08:21 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 08:15 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2025-10-08 ===
* 21:47 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76
* 21:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76
* 12:27 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]]
* 10:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 06:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71
* 06:43 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71

=== 2025-10-07 ===
* 18:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11
* 15:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69
* 15:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69
* 14:51 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69
* 14:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:58 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 08:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-10-06 ===
* 12:06 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7
* 11:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7
* 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 08:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 08:18 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76
* 07:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76

=== 2025-10-03 ===
* 12:51 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-6
* 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-6
* 12:45 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-5
* 09:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-5

=== 2025-10-02 ===
* 13:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-haproxy-7
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-haproxy-7
* 13:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 09:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55
* 08:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55

=== 2025-10-01 ===
* 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-09-30 ===
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67
* 08:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67

=== 2025-09-29 ===
* 13:23 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 13:23 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 11:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:34 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 10:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79
* 07:00 godog: kick stuck nfs workers from clouddumps1001

=== 2025-09-28 ===
* 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:13 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]])
* 08:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]])
* 08:10 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]])
* 08:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:07 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 08:07 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-09-25 ===
* 18:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 15:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 11:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-09-24 ===
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 20:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 17:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 17:30 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43
* 17:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 17:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43
* 17:07 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43
* 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]])
* 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]])
* 13:49 dcaro: patched all tools with new resource defaults, everything looks good
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:09 dcaro: depolyed jobs-api change to default resources, patching existing jobs
* 13:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 12:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 03:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12

=== 2025-09-23 ===
* 20:08 andrewbogott: creating puppetdbpostgres and adding it to tools-puppetdb-2 to store postgres data; the root volume of that VM was filling up and causing widespread puppet issues
* 01:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-09-22 ===
* 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-09-21 ===
* 09:17 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2
* 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2
* 03:16 dcaro: acking and silencing CPU capacity alerts to handle on Monday, they should not page
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 01:46 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-113.tools.eqiad1.wikimedia.cloud to the cluster
* 01:36 andrewbogott: adding additional worker node in response to repeated capacity alerts
* 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2025-09-19 ===
* 13:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 13:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11

=== 2025-09-18 ===
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools)
* 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools)
* 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 09:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 08:52 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3
* 08:34 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3
* 06:47 taavi: delete tools-sgebastion-10 [[phab:T314665|T314665]]

=== 2025-09-17 ===
* 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32
* 12:53 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32
* 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools)
* 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools)
* 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 08:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10
* 08:08 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10

=== 2025-09-16 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:57 taavi: delete tools-sgebastion puppet prefix [[phab:T314665|T314665]]
* 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 taavi: shutdown tools-sgebastion-10 [[phab:T314665|T314665]]
* 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-13
* 14:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-13
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-12
* 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-12
* 07:11 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75
* 06:57 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75
* 06:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 06:51 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-09-15 ===
* 16:22 taavi: reboot old bastions to kick long-living connections into newer ones
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 12:47 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66
* 12:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66

=== 2025-09-12 ===
* 08:49 taavi: pointing login.toolforge.org to tools-bastion-15 [[phab:T392510|T392510]]
* 08:33 taavi: pointing dev.toolforge.org to tools-bastion-14 [[phab:T392510|T392510]]
* 07:14 godog: uncordon tools-k8s-worker-nfs-53 after failed cookbook (?) yesterday

=== 2025-09-11 ===
* 14:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:36 godog: drain/reboot tools-k8s-worker-nfs-46 - [[phab:T404322|T404322]]
* 14:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:22 andrewbogott: actually I didn't drain tools-k8s-worker-nfs-53 because the alert cleared on its own
* 14:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-53
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53
* 14:21 andrewbogott: draining/rebooting tools-k8s-worker-nfs-53 because of procs in D state
* 13:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53
* 13:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-14.tools.eqiad1.wikimedia.cloud
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-14.tools.eqiad1.wikimedia.cloud
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-15.tools.eqiad1.wikimedia.cloud
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-15.tools.eqiad1.wikimedia.cloud
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-09-10 ===
* 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T403964|T403964]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:45 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:31 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers ([[phab:T403964|T403964]])
* 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]])

=== 2025-09-09 ===
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:55 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) ([[phab:T404047|T404047]])
* 08:55 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot ([[phab:T404047|T404047]])

=== 2025-09-08 ===
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 15:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:16 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97)
* 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 12:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]])
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]])
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]])
* 11:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]])
* 11:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 10:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 10:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]])
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]])
* 09:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]])
* 09:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]])
* 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:40 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 09:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 08:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 08:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]])
* 08:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]])
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97)
* 08:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests

=== 2025-09-06 ===
* 23:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35
* 23:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35

=== 2025-09-05 ===
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-09-04 ===
* 19:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:52 dcaro: added 'disable-ssl' to tools replica.my.cnf

=== 2025-09-03 ===
* 17:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 09:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-09-02 ===
* 17:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 17:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-29 ===
* 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting

=== 2025-08-28 ===
* 16:52 taavi: rebuild tcl, mariadb images on top of trixie [[phab:T400256|T400256]]
* 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-08-27 ===
* 18:05 taavi: copy missing aptly packages to trixie-&lt;nowiki&gt;{&lt;/nowiki&gt;tools,toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;
* 11:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-08-26 ===
* 13:42 dcaro: extended object storage quota to 100G ([[phab:T402923|T402923]])
* 10:25 dhinus: shut down tools-harbor-1 (no longer used)

=== 2025-08-25 ===
* 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-81
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-81

=== 2025-08-21 ===
* 12:28 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:31 godog: reboot nfs workers to reset processes stuck in D state
* 07:28 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 04:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80
* 03:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80

=== 2025-08-20 ===
* 08:09 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]])

=== 2025-08-19 ===
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 15:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 15:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:50 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:37 dcaro: flipped the tools-harbor.wmcloud.org endpoint to point to tools-harbor-2 ([[phab:T350687|T350687]])
* 14:22 Raymond_Ndibe: setting tools-harbor-1 as read-only ([[phab:T350687|T350687]])
* 13:24 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 13:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-18 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]])
* 17:49 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.16.0 ([[phab:T362869|T362869]])
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]])
* 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]])
* 17:48 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/metrics-server:v0.7.2 ([[phab:T362869|T362869]])
* 17:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]])
* 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]])
* 16:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]])
* 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-harbor-2.tools.eqiad1.wikimedia.cloud
* 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud

=== 2025-08-16 ===
* 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-111
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-111
* 21:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-111
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-111

=== 2025-08-15 ===
* 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 19:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103

=== 2025-08-14 ===
* 15:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 15:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41
* 11:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41
* 11:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found
* 11:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found
* 02:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-13 ===
* 16:36 dcaro: reverting jobs-api release ([[phab:T401846|T401846]])
* 11:18 taavi: delete tools-prometheus-6, shutdown for a while
* 08:51 godog: bounce stashbot
* 08:33 godog: refresh machine-id on tools-k8s-worker-[102-103,105-112].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-3,5,7-14,16-17,19,21-24,26-27,32-48,50,53-55 ,57-58,61,65-82].tools.eqiad1.wikimedia.cloud

=== 2025-08-12 ===
* 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config
* 15:34 taavi: building initial trixie based images [[phab:T400255|T400255]]
* 12:50 dcaro: redepoly kyverno ([[phab:T394787|T394787]])
* 12:49 dcaro: manually migrate cleanuppolicies.kyverno.io and clustercleanuppolicies.kyverno.io (using kyverno cli) ([[phab:T394787|T394787]])
* 10:01 dcaro: starting upgrade for kyverno ([[phab:T394787|T394787]])
* 10:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli

=== 2025-08-11 ===
* 12:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud
* 08:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud
* 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58
* 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58

=== 2025-08-08 ===
* 06:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-07 ===
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67
* 14:20 andrewbogott: draining and rebooting tools-k8s-worker-nfs-67
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-06 ===
* 17:54 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 17:53 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests
* 17:41 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]])
* 17:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]])
* 17:39 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]])
* 17:38 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]])
* 17:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-08-05 ===
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
* 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24
* 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24
* 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 03:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 02:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 02:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 02:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 02:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 02:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 02:30 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 02:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 01:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 01:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-08-04 ===
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]])
* 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]])
* 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2
* 11:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2

=== 2025-08-01 ===
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli

=== 2025-07-31 ===
* 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 16:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 04:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 04:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 04:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 04:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 04:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 04:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 04:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 04:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 04:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-07-30 ===
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-29 ===
* 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 15:56 andrewbogott: draining and restarting  tools-k8s-worker-nfs-74
* 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32
* 15:32 andrewbogott: draining and restarting  tools-k8s-worker-nfs-58 and  tools-k8s-worker-nfs-32
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:16 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-82.tools.eqiad1.wikimedia.cloud to the cluster
* 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:06 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:05 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-81.tools.eqiad1.wikimedia.cloud to the cluster
* 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.quota_increase
* 12:39 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:31 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-80.tools.eqiad1.wikimedia.cloud to the cluster
* 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:00 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:54 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:28 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:07 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:59 wmbot~dcaro@acme: Added a new k8s worker tools-k8s-worker-112.tools.eqiad1.wikimedia.cloud to the cluster
* 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 08:49 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:14 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2025-07-28 ===
* 20:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 19:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 11:58 taavi: update pywikibot image to 10.2.0 [[phab:T396933|T396933]]

=== 2025-07-26 ===
* 07:16 wmbot~root@toolforge: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:16 wmbot~root@toolforge: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli

=== 2025-07-23 ===
* 18:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-07-21 ===
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-07-19 ===
* 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 13:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-07-18 ===
* 10:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44

=== 2025-07-14 ===
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78

=== 2025-07-13 ===
* 03:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12

=== 2025-07-11 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37
* 09:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37

=== 2025-07-09 ===
* 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli
* 10:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli
* 09:55 dcaro: adding arch arm64 to all toolforge repos ([[phab:T398016|T398016]])
* 09:40 dcaro: added arch arm64 to jessie-tools repo ([[phab:T398016|T398016]])

=== 2025-07-08 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-07 ===
* 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53
* 17:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53
* 16:39 dcaro: pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest
* 16:05 dcaro: clearing images from tools-imagebuilder-2 as it's out of space
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-06 ===
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8

=== 2025-07-05 ===
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57
* 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs

=== 2025-07-04 ===
* 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-07-03 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging

=== 2025-07-02 ===
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55
* 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55
* 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-07-01 ===
* 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging
* 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging
* 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-06-30 ===
* 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70
* 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]])
* 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]])
* 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]])
* 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]])

=== 2025-06-28 ===
* 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67
* 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67
* 10:08 dcaro: left a tmux running with a script to restart nginx if stuck
* 09:59 dcaro: restarted nginx in tools-static

=== 2025-06-27 ===
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46

=== 2025-06-26 ===
* 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-25 ===
* 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38

=== 2025-06-24 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-23 ===
* 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]]

=== 2025-06-22 ===
* 00:09 andrewbogott: rebooting tools-prometheus-8

=== 2025-06-21 ===
* 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of  D state
* 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12
* 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-19 ===
* 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools

=== 2025-06-18 ===
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable

=== 2025-06-16 ===
* 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39

=== 2025-06-14 ===
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-06-12 ===
* 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh)
* 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-06-11 ===
* 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1
* 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1
* 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1

=== 2025-06-10 ===
* 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved

=== 2025-06-06 ===
* 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74
* 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-06-05 ===
* 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert
* 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-30 ===
* 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11
* 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 07:38 taavi: reboot tools-static-15 to unstuck NFS things

=== 2025-05-24 ===
* 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65
* 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65

=== 2025-05-23 ===
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65
* 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43
* 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43

=== 2025-05-22 ===
* 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55
* 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21
* 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 18:15 dcaro: restart tools-static nginx due to nfs hiccup
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7
* 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1
* 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]]

=== 2025-05-21 ===
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud
* 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh)
* 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead

=== 2025-05-20 ===
* 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6
* 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35
* 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6
* 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6
* 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6
* 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6
* 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97)
* 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6
* 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown
* 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud
* 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-19 ===
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]])
* 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-14 ===
* 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36
* 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36

=== 2025-05-12 ===
* 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  ([[phab:T393686|T393686]])
* 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19

=== 2025-05-10 ===
* 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo&lt;nowiki&gt;{&lt;/nowiki&gt;,.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]])
* 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.service &amp;&amp; systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # try to reset the rate limits this way ([[phab:T393732|T393732]])
* 16:22 lucaswerkmeister: systemctl restart sssd-&lt;nowiki&gt;{&lt;/nowiki&gt;pam&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;,sudo&lt;nowiki&gt;}&lt;/nowiki&gt;.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]?
* 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient)
* 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam&lt;nowiki&gt;{&lt;/nowiki&gt;,&lt;nowiki&gt;{&lt;/nowiki&gt;,-priv&lt;nowiki&gt;}&lt;/nowiki&gt;.socket&lt;nowiki&gt;}&lt;/nowiki&gt; # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]?

=== 2025-05-09 ===
* 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake
* 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory)
* 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs

=== 2025-05-08 ===
* 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission
* 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state
* 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-07 ===
* 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
* 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:33 dcaro: released jobs-cli 16.1.12
* 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69

=== 2025-05-04 ===
* 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06)

=== 2025-05-02 ===
* 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]]
* 13:55 taavi: reboot tools-static-15

=== 2025-04-28 ===
* 13:07 dhinus: tools-db-4: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:06 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T392596|T392596]]
* 13:05 dhinus: tools-db-5: systemctl stop mariadb &amp;&amp; systemctl start mariadb [[phab:T318479|T318479]]

=== 2025-04-24 ===
* 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12
* 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing &quot;sudo: PAM account management error: Authentication service cannot retrieve authentication info&quot;
* 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 11:51 taavi: add missing ICMPv6 security group rule to 'default' group
* 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]]

=== 2025-04-23 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58
* 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud
* 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles

=== 2025-04-21 ===
* 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]]

=== 2025-04-17 ===
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-11 ===
* 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-10 ===
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76

=== 2025-04-09 ===
* 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff)
* 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-04-07 ===
* 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]])
* 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-06 ===
* 02:12 andrewbogott: truncating large logfiles on tools nfs

=== 2025-04-04 ===
* 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes

=== 2025-04-03 ===
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33
* 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33
* 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13

=== 2025-04-02 ===
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55
* 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48

=== 2025-04-01 ===
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24

=== 2025-03-31 ===
* 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
* 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
* 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74

=== 2025-03-28 ===
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes
* 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]])
* 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]])
* 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]])
* 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]])

=== 2025-03-27 ===
* 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33
* 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers
* 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster
* 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72
* 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72

=== 2025-03-25 ===
* 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2
* 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2
* 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72

=== 2025-03-22 ===
* 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-18 ===
* 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh
* 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])
* 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]])

=== 2025-03-17 ===
* 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]])
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]])

=== 2025-03-16 ===
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38

=== 2025-03-15 ===
* 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]])
* 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]]

=== 2025-03-13 ===
* 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44

=== 2025-03-12 ===
* 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o
* 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-03-07 ===
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2025-03-06 ===
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-03-05 ===
* 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values)
* 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]])
* 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes
* 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-03-04 ===
* 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]])
* 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]])
* 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual)
* 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-03 ===
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-03-01 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57

=== 2025-02-27 ===
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder

=== 2025-02-26 ===
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-25 ===
* 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58

=== 2025-02-24 ===
* 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-21 ===
* 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75
* 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75

=== 2025-02-20 ===
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-19 ===
* 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55
* 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37

=== 2025-02-18 ===
* 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54
* 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]])
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]])

=== 2025-02-17 ===
* 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-10 ===
* 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-02-09 ===
* 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes

=== 2025-02-07 ===
* 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]]
* 17:58 andrewbogott: &quot;SET GLOBAL read_only=OFF;  &quot; on tools-db-4; both -5 and -4 were set to read only.  No idea why or how...
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07
* 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07

=== 2025-02-06 ===
* 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage
* 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-03 ===
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6
* 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9
* 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7

=== 2025-02-01 ===
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108
* 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107
* 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106
* 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105
* 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103
* 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102
* 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]]
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43
* 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
* 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32
* 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12
* 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11
* 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]]
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11
* 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10
* 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10

=== 2025-01-31 ===
* 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]]

=== 2025-01-29 ===
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-27 ===
* 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-26 ===
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44
* 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster
* 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster
* 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]])
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]])
* 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55
* 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55

=== 2025-01-24 ===
* 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41

=== 2025-01-23 ===
* 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs

=== 2025-01-22 ===
* 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2025-01-18 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2025-01-17 ===
* 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing)

=== 2025-01-15 ===
* 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-13 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]])
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]])
* 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]])
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]])
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]])
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]])
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]])
* 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]])
* 20:49 dcaro: restart prometheus to pick up the new ips for vms and such
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]])
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]])
* 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]])
* 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots

=== 2025-01-12 ===
* 09:53 taavi: hard reboot tools-k8s-worker-nfs-55

=== 2025-01-08 ===
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]])
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]])
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]])
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]])
* 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]])
* 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]])
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]])
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]])
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]])
* 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]])
* 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]])
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]])
* 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]])
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]])
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]])
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]])
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]])
* 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38
* 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42
* 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22
* 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70
* 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70
* 14:16 dcaro: reboot tools-static-15 nfs is stuck

=== 2025-01-07 ===
* 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor

=== 2025-01-06 ===
* 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2025-01-05 ===
* 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13

=== 2025-01-03 ===
* 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69
* 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69
* 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69
* 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69

=== 2025-01-02 ===
* 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61

=== 2025-01-01 ===
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34
* 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34

=== 2024-12-13 ===
* 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68
* 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44
* 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73
* 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73

=== 2024-12-12 ===
* 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5
* 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5

=== 2024-12-06 ===
* 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]])
* 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]])
* 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]])
* 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing
* 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main
* 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main

=== 2024-11-29 ===
* 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-27 ===
* 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns  # update resolv.conf in coredns containers

=== 2024-11-26 ===
* 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7
* 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9
* 10:22 dcaro: rebooting k8s-control-9
* 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8
* 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8
* 10:17 dcaro: rebooting k8s-control-8
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72
* 09:14 dcaro: restarting tools-k8s-worker-nfs-72
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70
* 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 09:12 dcaro: restarting tools-k8s-worker-nfs-70
* 09:11 dcaro: restarting tools-k8s-worker-nfs-50
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61
* 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]])
* 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]])

=== 2024-11-25 ===
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2024-11-23 ===
* 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]])
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]])
* 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-18 ===
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-15 ===
* 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]])
* 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]])
* 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])

=== 2024-11-14 ===
* 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]])
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]])
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-10 ===
* 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]])
* 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]])

=== 2024-11-06 ===
* 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]])
* 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:42 dcaro: added api.svc.toolforge.org dns record entry
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-22 ===
* 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
* 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
* 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]]

=== 2024-10-16 ===
* 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-14 ===
* 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]])
* 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
* 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-10-09 ===
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
* 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating
* 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]]
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-04 ===
* 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-02 ===
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-01 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 10:28 dcaro: updated ci image with latest precommit versions
* 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
* 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-30 ===
* 18:25 taavi: run striker migrations [[phab:T359428|T359428]]

=== 2024-09-28 ===
* 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
* 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
* 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

=== 2024-09-24 ===
* 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-20 ===
* 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]])
* 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]])
* 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]])
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]])
* 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]])
* 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]])
* 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]])
* 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]])
* 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
* 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]])
* 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-17 ===
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]])
* 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]])
* 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
* 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
* 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
* 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
* 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
* 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
* 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
* 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
* 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
* 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
* 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
* 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
* 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
* 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
* 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
* 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
* 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]])
* 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]])

=== 2024-09-16 ===
* 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
* 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
* 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-09-13 ===
* 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]])
* 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]])

=== 2024-09-12 ===
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])
* 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]])

=== 2024-09-11 ===
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-09 ===
* 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

=== 2024-09-06 ===
* 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
* 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-09-05 ===
* 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]])
* 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]])
* 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]])
* 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]])
* 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])
* 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]])
* 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-03 ===
* 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]])
* 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]])
* 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]])
* 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
* 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
* 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
* 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

=== 2024-09-02 ===
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
* 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
* 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
* 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
* 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
* 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
* 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
* 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
* 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
* 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
* 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
* 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
* 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]])
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
* 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
* 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
* 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
* 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
* 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
* 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
* 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
* 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
* 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
* 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]])
* 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]])
* 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]])
* 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]])
* 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]])
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-08-26 ===
* 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
* 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
* 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
* 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]])
* 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]])
* 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
* 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:18 dcaro: scale up cordens deployment to 4 replicas

=== 2024-08-21 ===
* 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-19 ===
* 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
* 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
* 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

=== 2024-08-15 ===
* 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
* 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

=== 2024-08-13 ===
* 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
* 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

=== 2024-08-12 ===
* 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-06 ===
* 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
* 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-08-05 ===
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
* 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
* 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
* 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
* 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]]
* 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
* 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

=== 2024-07-30 ===
* 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

=== 2024-07-29 ===
* 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
* 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
* 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
* 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
* 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 17:00 dcaro: moving the toolforge apt repo to tools-services-06
* 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
* 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-19 ===
* 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
* 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

=== 2024-07-18 ===
* 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
* 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
* 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
* 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
* 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
* 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
* 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
* 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
* 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
* 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
* 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
* 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
* 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
* 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
* 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
* 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
* 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
* 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
* 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
* 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
* 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
* 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
* 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
* 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
* 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
* 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
* 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
* 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
* 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
* 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
* 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
* 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
* 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
* 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
* 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
* 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
* 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
* 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
* 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-11 ===
* 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
* 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
* 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
* 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
* 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
* 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
* 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
* 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
* 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]]
* 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]])
* 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]])
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
* 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
* 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]])
* 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]]
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

=== 2024-06-25 ===
* 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-&gt; dict[str, Any]` type annotations causing [[phab:T368463|T368463]]
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
* 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
* 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
* 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
* 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
* 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
* 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
* 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
* 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
* 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
* 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
* 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
* 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
* 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
* 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
* 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
* 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
* 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
* 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
* 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
* 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
* 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
* 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
* 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
* 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
* 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
* 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
* 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
* 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
* 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
* 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
* 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
* 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
* 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
* 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

=== 2024-06-24 ===
* 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
* 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
* 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
* 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
* 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
* 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
* 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
* 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
* 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
* 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
* 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
* 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
* 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
* 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
* 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
* 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
* 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
* 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
* 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
* 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
* 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
* 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
* 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
* 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
* 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
* 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
* 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
* 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
* 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
* 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
* 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
* 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
* 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
* 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
* 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
* 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
* 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
* 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
* 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
* 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
* 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
* 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
* 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
* 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
* 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
* 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
* 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
* 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
* 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
* 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
* 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
* 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
* 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
* 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
* 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
* 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
* 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
* 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
* 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
* 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
* 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

=== 2024-06-20 ===
* 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]]
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-14 ===
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]].
* 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 13:45 taavi: hard reboot tools-k8s-control-7
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
* 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:57 dcaro: cleaning old maintain-kubeusers configmaps
* 10:45 dcaro: cleaning up old resourcequotas

=== 2024-06-10 ===
* 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

=== 2024-06-07 ===
* 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 dcaro: deploying toolforge-webservice 0.103.6
* 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
* 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
* 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
* 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
* 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
* 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
* 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-05-29 ===
* 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
* 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-28 ===
* 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-25 ===
* 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-22 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

=== 2024-05-15 ===
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]])
* 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
* 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

=== 2024-05-07 ===
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-05 ===
* 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]]

=== 2024-04-24 ===
* 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-17 ===
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
* 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
* 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
* 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

=== 2024-04-16 ===
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

=== 2024-04-15 ===
* 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-11 ===
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-09 ===
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
* 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

=== 2024-04-08 ===
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
* 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
* 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
* 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
* 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

=== 2024-04-05 ===
* 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
* 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
* 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]]

=== 2024-04-02 ===
* 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]])
* 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]])
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

=== 2024-03-28 ===
* 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
* 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]]
* 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
* 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
* 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
* 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
* 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

=== 2024-03-27 ===
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

=== 2024-03-26 ===
* 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
* 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
* 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
* 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
* 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
* 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]]

=== 2024-03-25 ===
* 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
* 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
* 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-22 ===
* 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

=== 2024-03-21 ===
* 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
* 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
* 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
* 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
* 12:30 taavi: move checker service address to tools-checker-5
* 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
* 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
* 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

=== 2024-03-19 ===
* 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
* 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
* 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]]
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]])

=== 2024-03-18 ===
* 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 taavi: restart harbor services after docker service restart
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]]
* 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]])
* 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
* 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]]

=== 2024-03-15 ===
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
* 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
* 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
* 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]]
* 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]]

=== 2024-03-13 ===
* 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

=== 2024-03-12 ===
* 12:38 taavi: hard reboot tools-prometheus-6
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-11 ===
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]]

=== 2024-03-09 ===
* 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

=== 2024-03-08 ===
* 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-07 ===
* 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-06 ===
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
* 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

=== 2024-03-05 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]])
* 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

=== 2024-03-04 ===
* 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

=== 2024-03-03 ===
* 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-29 ===
* 14:36 dcaro: deploy webservice 0.103.3

=== 2024-02-28 ===
* 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]])
* 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])

=== 2024-02-23 ===
* 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
* 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]]

=== 2024-02-22 ===
* 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]])
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]])
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
* 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]])
* 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
* 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
* 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
* 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
* 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

=== 2024-02-21 ===
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
* 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
* 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

=== 2024-02-20 ===
* 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
* 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
* 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
* 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
* 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
* 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
* 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
* 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
* 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
* 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
* 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
* 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
* 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
* 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
* 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
* 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
* 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
* 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
* 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
* 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
* 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
* 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
* 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
* 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

=== 2024-02-19 ===
* 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]])
* 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
* 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
* 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
* 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
* 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
* 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
* 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
* 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
* 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-16 ===
* 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
* 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
* 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
* 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
* 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
* 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
* 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

=== 2024-02-15 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
* 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
* 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
* 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
* 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
* 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
* 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

=== 2024-02-14 ===
* 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
* 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
* 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
* 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
* 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
* 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
* 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
* 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
* 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
* 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
* 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
* 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
* 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
* 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
* 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
* 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
* 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
* 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

=== 2024-02-13 ===
* 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
* 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
* 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
* 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
* 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
* 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

=== 2024-02-12 ===
* 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
* 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
* 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
* 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
* 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
* 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
* 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
* 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
* 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
* 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
* 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
* 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
* 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
* 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
* 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
* 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
* 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-11 ===
* 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-02-09 ===
* 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
* 13:06 taavi: reboot tools-sgecron-2 due to high load
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
* 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
* 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
* 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
* 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

=== 2024-02-08 ===
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
* 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
* 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
* 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
* 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
* 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
* 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
* 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
* 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

=== 2024-02-07 ===
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
* 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
* 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
* 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
* 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
* 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

=== 2024-02-06 ===
* 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-01-31 ===
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-30 ===
* 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
* 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
* 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
* 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
* 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
* 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
* 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
* 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
* 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
* 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
* 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]])
* 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]])
* 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]]
* 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]])
* 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]])
* 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]])

=== 2024-01-29 ===
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
* 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
* 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
* 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
* 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
* 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
* 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
* 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
* 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
* 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
* 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
* 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
* 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
* 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
* 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
* 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
* 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-26 ===
* 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

=== 2024-01-25 ===
* 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

=== 2024-01-23 ===
* 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
* 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
* 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-18 ===
* 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
* 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

=== 2024-01-17 ===
* 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]]
* 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]])
* 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]])
* 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]]

=== 2024-01-15 ===
* 09:18 taavi: reboot stuck tools-k8s-worker-84

=== 2024-01-12 ===
* 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
* 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

=== 2024-01-11 ===
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-10 ===
* 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:17 taavi: reboot tools-k8s-worker-98

=== 2024-01-09 ===
* 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor  -- [[phab:T354714|T354714]]
* 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
* 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
* 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
* 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

=== 2024-01-08 ===
* 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:17 taavi: reboot tools-sgeexec-10-21

=== 2024-01-05 ===
* 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2024-01-04 ===
* 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

=== 2024-01-03 ===
* 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
* 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

=== 2024-01-02 ===
* 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]])
* 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]])
* 10:37 dcaro: hard reboot tools-harbor-1
* 10:13 dhinus: hard reboot tools-harbor-1

=== 2024-01-01 ===
* 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]]

==Archives==
* [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014)
* [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017)
* [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019)
* [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021)
* [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023)

&lt;/noinclude&gt;
{{SAL|Project Name=tools}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>6s00g7ef4spgowcpqd1c1yktlk7nwwp</sha1>
    </revision>
  </page>
  <page>
    <title>Server Admin Log</title>
    <ns>0</ns>
    <id>7919</id>
    <revision>
      <id>2435522</id>
      <parentid>2435488</parentid>
      <timestamp>2026-07-12T02:00:13Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image</comment>
      <origin>2435522</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="329829" sha1="g73lyduu6muibg4iyr02spptgh67h3s" xml:space="preserve">
== 2026-07-12 ==
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-11 ==
* 02:06 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 26s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-10 ==
* 19:12 jhathaway@dns1004: END - running authdns-update
* 19:10 jhathaway@dns1004: START - running authdns-update
* 18:23 mutante: vrts2002 rebooting (not the active host)
* 18:21 mutante: lists2001, phab2003 - rebooting (not the active hosts)
* 18:16 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic2-codfw
* 18:15 swfrench@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic2-codfw
* 17:15 mutante: [doc1004:~] $ sudo systemctl start rsync-doc-host-data-sync ([[phab:T431856|T431856]])
* 17:09 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1004.eqiad.wmnet
* 17:08 jhathaway@dns1004: END - running authdns-update
* 17:07 jhathaway@dns1004: START - running authdns-update
* 17:06 jhathaway: depooling puppetserver1002, cause of errors is still unknown
* 17:03 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1004.eqiad.wmnet
* 16:57 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1003.eqiad.wmnet
* 16:51 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1003.eqiad.wmnet
* 16:48 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 16:48 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2004.codfw.wmnet
* 16:42 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2004.codfw.wmnet
* 16:41 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2003.codfw.wmnet
* 16:35 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2003.codfw.wmnet
* 16:33 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2002.codfw.wmnet
* 16:27 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2002.codfw.wmnet
* 16:25 mutante: gitlab-runners (production) rebooting cluster one by one
* 16:17 mutante: etherpad1004/etherpad2002 - (etherpad.wikimedia.org) - rebooting
* 16:13 mutante: doc1004/doc2003 (doc.wikimedia.org backends) - rebooting
* 16:02 mutante: releases1003/releases2003 (releases.wikimedia.org backends) - rebooting for maintenance
* 15:26 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2007-dev.codfw.wmnet
* 15:19 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet
* 15:14 filippo@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcephosd2007-dev.codfw.wmnet
* 15:14 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet
* 15:14 filippo@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudcephosd2006-dev.codfw.wmnet
* 15:07 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2006-dev.codfw.wmnet
* 15:07 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2005-dev.codfw.wmnet
* 14:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2005-dev.codfw.wmnet
* 14:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet
* 14:53 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2004-dev.codfw.wmnet
* 14:53 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2007-dev.codfw.wmnet
* 14:51 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1054.eqiad.wmnet
* 14:51 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet
* 14:51 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet
* 14:47 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2007-dev.codfw.wmnet
* 14:47 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet
* 14:41 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2006-dev.codfw.wmnet
* 14:40 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet
* 14:37 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2005-dev.codfw.wmnet
* 14:37 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2005-dev.codfw.wmnet
* 14:29 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2005-dev.codfw.wmnet
* 14:29 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2006-dev.codfw.wmnet
* 14:21 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2006-dev.codfw.wmnet
* 14:21 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet
* 14:15 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2010-dev.codfw.wmnet
* 14:15 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2004-dev.codfw.wmnet
* 14:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1054.eqiad.wmnet with OS trixie
* 14:09 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudgw2004-dev.codfw.wmnet
* 14:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2003-dev.codfw.wmnet
* 14:02 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudgw2003-dev.codfw.wmnet
* 14:02 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet
* 13:53 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudlb2004-dev.codfw.wmnet
* 13:53 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2003-dev.codfw.wmnet
* 13:48 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 13:44 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudlb2003-dev.codfw.wmnet
* 13:44 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2002-dev.codfw.wmnet
* 13:42 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 13:41 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 13:41 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 13:37 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudlb2002-dev.codfw.wmnet
* 13:37 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudidp2001-dev.codfw.wmnet
* 13:33 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
* 13:33 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudidp2001-dev.codfw.wmnet
* 13:33 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
* 13:26 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudnet2006-dev.codfw.wmnet
* 13:26 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
* 13:23 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
* 13:18 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudnet2005-dev.codfw.wmnet
* 13:18 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2005-dev.codfw.wmnet
* 13:12 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudservices2005-dev.codfw.wmnet
* 13:11 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2004-dev.codfw.wmnet
* 13:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudservices2004-dev.codfw.wmnet
* 13:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudweb2002-dev.wikimedia.org
* 13:05 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:05 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1054
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1054
* 13:04 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1054
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1054.eqiad.wmnet 49.32.64.10.in-addr.arpa 9.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:04 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1054.eqiad.wmnet 49.32.64.10.in-addr.arpa 9.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1054 - blake@cumin1003&quot;
* 13:04 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1054 - blake@cumin1003&quot;
* 13:01 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudweb2002-dev.wikimedia.org
* 13:00 blake@cumin1003: START - Cookbook sre.dns.netbox
* 12:59 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1054
* 12:57 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1054.eqiad.wmnet with OS trixie
* 12:57 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet
* 12:56 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet
* 12:56 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1054.eqiad.wmnet
* 12:47 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS trixie
* 12:44 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:39 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 12:39 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 12:38 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 12:37 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 12:14 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:10 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:09 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:09 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:09 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:08 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:07 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:00 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:00 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 11:51 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 11:49 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 11:48 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:47 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:44 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:32 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet
* 11:32 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1053.eqiad.wmnet
* 11:32 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet
* 11:32 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet
* 11:32 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2001.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2001.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;wikikube-worker2001*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
* 11:30 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:30 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:21 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 18 hosts with reason: reboot &amp; upgrade
* 11:20 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS trixie
* 11:16 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:15 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:14 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:14 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 11 hosts
* 11:14 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 11 hosts
* 11:13 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:08 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 11:02 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1053.eqiad.wmnet with OS trixie
* 11:01 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 10:58 cgoubert@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 10:57 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS trixie
* 10:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
* 10:38 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
* 10:38 cgoubert@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;wikikube-worker2001*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
* 10:35 topranks: adjust IBGP outbound policy on lsw1-e2-codfw [[phab:T423430|T423430]] towards ssw1-e1-codfw
* 10:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 cgoubert@cumin1003: START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:27 cgoubert@cumin1003: END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on A:wikikube-worker-codfw
* 10:27 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS bookworm
* 10:25 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 10:24 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 10:24 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 10:15 cgoubert@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 10:11 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 10:11 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 10:08 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 10:08 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 10:07 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 10:06 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 10:00 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 09:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS bookworm
* 09:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2005-2006,2011-2012].codfw.wmnet
* 09:55 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 09:52 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2005-2006,2011-2012].codfw.wmnet
* 09:51 cgoubert@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on A:wikikube-worker-codfw
* 09:41 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage
* 09:37 topranks: apply new IBGP outbound policy on lsw1-e2-codfw [[phab:T423430|T423430]]
* 09:36 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:36 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage
* 09:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1053
* 09:16 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1053
* 09:15 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1053
* 09:15 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1053.eqiad.wmnet 48.32.64.10.in-addr.arpa 8.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:15 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1053.eqiad.wmnet 48.32.64.10.in-addr.arpa 8.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:15 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:15 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1053 - blake@cumin1003&quot;
* 09:15 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1053 - blake@cumin1003&quot;
* 09:11 blake@cumin1003: START - Cookbook sre.dns.netbox
* 09:11 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1053
* 09:08 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1053.eqiad.wmnet with OS trixie
* 09:08 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet
* 09:08 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet
* 09:08 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1053.eqiad.wmnet
* 09:04 brouberol@dns1004: END - running authdns-update
* 09:03 brouberol@dns1004: START - running authdns-update
* 08:41 javiermonton@deploy2003: Finished deploy [analytics/refinery@1abf22e] (thin): Regular analytics weekly train THIN [analytics/refinery@1abf22ea] (duration: 02m 11s)
* 08:38 javiermonton@deploy2003: Started deploy [analytics/refinery@1abf22e] (thin): Regular analytics weekly train THIN [analytics/refinery@1abf22ea]
* 08:38 javiermonton@deploy2003: Finished deploy [analytics/refinery@1abf22e]: Regular analytics weekly train [analytics/refinery@1abf22ea] (duration: 05m 17s)
* 08:38 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:34 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 08:33 javiermonton@deploy2003: Started deploy [analytics/refinery@1abf22e]: Regular analytics weekly train [analytics/refinery@1abf22ea]
* 08:32 javiermonton@deploy2003: Finished deploy [analytics/refinery@1abf22e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1abf22ea] (duration: 02m 03s)
* 08:30 javiermonton@deploy2003: Started deploy [analytics/refinery@1abf22e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1abf22ea]
* 08:30 JavierMonton: Deploying Refinery at {{Gerrit|1abf22ea}} for changes 1308121/T427068 1306491/T430020 and {{Gerrit|1308190}}
* 08:29 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 08:29 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 08:24 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 08:24 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 08:18 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 08:18 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 08:00 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet
* 08:00 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet
* 07:52 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:52 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:49 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 11 hosts with reason: reboot &amp; upgrade
* 07:47 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:47 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:44 dcausse@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:44 dcausse@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:23 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 10 hosts
* 07:23 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 10 hosts
* 06:45 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 10 hosts with reason: reboot &amp; upgrade
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-09 ==
* 23:33 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308671{{!}}Enable section share on Persian Wikipedia (T431514)]] (duration: 13m 26s)
* 23:29 ladsgroup@deploy2003: ladsgroup, jdlrobson: Continuing with deployment
* 23:22 ladsgroup@deploy2003: ladsgroup, jdlrobson: Backport for [[gerrit:1308671{{!}}Enable section share on Persian Wikipedia (T431514)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:20 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1308671{{!}}Enable section share on Persian Wikipedia (T431514)]]
* 22:57 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1165.eqiad.wmnet
* 22:56 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1165.eqiad.wmnet
* 22:56 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1165.eqiad.wmnet
* 22:45 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1165.eqiad.wmnet with OS trixie
* 22:38 arlolra@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:37 arlolra@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:37 arlolra@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:37 arlolra@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:37 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 22:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1165.eqiad.wmnet with reason: host reimage
* 22:17 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1165.eqiad.wmnet with reason: host reimage
* 22:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 22:12 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 22:04 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 22:04 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1165
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1165
* 22:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1165
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1165.eqiad.wmnet 115.48.64.10.in-addr.arpa 5.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 22:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1165.eqiad.wmnet 115.48.64.10.in-addr.arpa 5.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1165 - jasmine@cumin2002&quot;
* 22:02 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1165 - jasmine@cumin2002&quot;
* 22:02 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 21:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox
* 21:55 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1165
* 21:54 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1165.eqiad.wmnet with OS trixie
* 21:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 21:54 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1165.eqiad.wmnet
* 21:53 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 21:53 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1165.eqiad.wmnet
* 21:53 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1165.eqiad.wmnet
* 21:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 21:47 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 21:45 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 21:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage
* 21:43 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 21:43 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 21:42 maryum: Deploy fix for [[phab:T431684|T431684]]
* 21:38 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage
* 21:27 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309290{{!}}Introduce sharing of section function (T18691)]], [[gerrit:1309291{{!}}Drop the share icon from the mobile site (T18691)]] (duration: 34m 14s)
* 21:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1002
* 21:23 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1002
* 21:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS trixie
* 21:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 21:20 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 22s)
* 21:20 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 21:16 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 21:16 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 21:15 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 21:13 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2002.codfw.wmnet with OS bookworm
* 21:11 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1309290{{!}}Introduce sharing of section function (T18691)]], [[gerrit:1309291{{!}}Drop the share icon from the mobile site (T18691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
* 21:07 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 6 hosts with reason: reboots
* 20:54 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecycle work - bking@cumin2003
* 20:53 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:53 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1309290{{!}}Introduce sharing of section function (T18691)]], [[gerrit:1309291{{!}}Drop the share icon from the mobile site (T18691)]]
* 20:51 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 20:47 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecycle work - bking@cumin2003
* 20:42 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 20:41 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:41 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 20:40 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host relforge1008.eqiad.wmnet
* 20:40 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:37 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1009.eqiad.wmnet with OS trixie
* 20:33 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:32 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:32 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:31 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:31 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
* 20:29 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host relforge1008.eqiad.wmnet
* 20:24 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2002.codfw.wmnet with OS bookworm
* 20:23 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host relforge1008.eqiad.wmnet
* 20:23 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:23 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host relforge1008.eqiad.wmnet
* 20:22 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:22 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:21 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:21 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:20 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1009.eqiad.wmnet with reason: host reimage
* 20:16 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:15 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1009.eqiad.wmnet with reason: host reimage
* 20:12 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 20:02 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 19:55 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1009.eqiad.wmnet with OS trixie
* 19:54 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 19:43 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 19:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 19:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 19:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 19:25 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 18:42 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 18:41 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 18:16 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 18:15 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 17:45 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for doh5004.wikimedia.org
* 17:45 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for doh5004.wikimedia.org
* 17:38 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 17:35 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 17:29 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 17:26 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 17:09 mutante: zuul[12]00[123] - rebooting for maintenance
* 17:09 ebernhardson: start full in-place reindex of eqiad cirrussearch cluster
* 17:08 dzahn@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 17:08 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-cluster
* 17:03 ebernhardson: start full in-place reindex of codfw cirrussearch cluster
* 16:59 mutante: stewards1001/stewards2001 - reboot for maintenance
* 16:54 ebernhardson: start full in-place reindex of cloudelastic cluster
* 16:53 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 16:52 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 16:49 mutante: planet1003/planet2003 - rebooting
* 16:47 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on doh5004.wikimedia.org with reason: random high load, investigating
* 15:55 cjming@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 15:54 cjming@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 15:51 jynus: restarting backupmon1001
* 15:49 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 14 hosts
* 15:49 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 14 hosts
* 15:47 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backupmon1001.eqiad.wmnet with reason: restart
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:13 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:13 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:06 cjming@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 15:06 cjming@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 14:59 cjming@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
* 14:58 cjming@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
* 14:51 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 14 hosts
* 14:51 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 14 hosts
* 14:49 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 6 hosts with reason: reboot &amp; upgrade
* 14:48 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1002.eqiad.wmnet
* 14:48 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1002.eqiad.wmnet
* 14:47 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:45 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:42 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1052.eqiad.wmnet
* 14:42 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet
* 14:42 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet
* 14:32 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:32 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:31 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:31 elukey@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: sync
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:30 elukey@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: sync
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:28 elukey@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: sync
* 14:28 elukey@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: sync
* 14:26 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:23 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:23 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1052.eqiad.wmnet with OS trixie
* 14:19 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 6 hosts with reason: reboot &amp; upgrade
* 14:18 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:17 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:15 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:15 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:13 elukey: update druid indexation job for webrequest_sampled_live - [[phab:T427068|T427068]]
* 14:11 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:11 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:09 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating for papaul - jhancock@cumin2002&quot;
* 14:09 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating for papaul - jhancock@cumin2002&quot;
* 14:07 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:07 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:04 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cuminunpriv1001.eqiad.wmnet
* 13:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1052.eqiad.wmnet with reason: host reimage
* 13:57 moritzm: installing requests security updates
* 13:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host cuminunpriv1001.eqiad.wmnet
* 13:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:53 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1052.eqiad.wmnet with reason: host reimage
* 13:50 moritzm: installing python-cryptography security updates
* 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 13:44 Msz2001: UTC afternoon config+backport window is done
* 13:44 Msz2001: Updated `logging` on `metawiki` to fix log performers, [[phab:T431176|T431176]]#12105297
* 13:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 13:43 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 13:43 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org
* 13:41 mszwarc@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309101{{!}}Fix UserEditTracker spoiling ActorStore cache for cross-wiki lookups (T431176)]], [[gerrit:1309122{{!}}SuggestedInvestigations: Instrument link clicks in the cases table (T429320)]] (duration: 07m 30s)
* 13:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org
* 13:37 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1052
* 13:36 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1052
* 13:35 mszwarc@deploy2003: mszwarc: Backport for [[gerrit:1309101{{!}}Fix UserEditTracker spoiling ActorStore cache for cross-wiki lookups (T431176)]], [[gerrit:1309122{{!}}SuggestedInvestigations: Instrument link clicks in the cases table (T429320)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:35 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1052
* 13:35 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1052.eqiad.wmnet 47.32.64.10.in-addr.arpa 7.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:35 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1052.eqiad.wmnet 47.32.64.10.in-addr.arpa 7.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:35 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:35 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1052 - blake@cumin1003&quot;
* 13:35 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1052 - blake@cumin1003&quot;
* 13:34 mszwarc@deploy2003: Started scap sync-world: Backport for [[gerrit:1309101{{!}}Fix UserEditTracker spoiling ActorStore cache for cross-wiki lookups (T431176)]], [[gerrit:1309122{{!}}SuggestedInvestigations: Instrument link clicks in the cases table (T429320)]]
* 13:31 blake@cumin1003: START - Cookbook sre.dns.netbox
* 13:31 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1052
* 13:30 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1052.eqiad.wmnet with OS trixie
* 13:30 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet
* 13:29 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet
* 13:29 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1052.eqiad.wmnet
* 13:17 jforrester@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309112{{!}}abstractwiki: Don't run our jobs too quickly, they'll collide and error (T430898)]] (duration: 11m 26s)
* 13:13 jforrester@deploy2003: jforrester: Continuing with deployment
* 13:08 jforrester@deploy2003: jforrester: Backport for [[gerrit:1309112{{!}}abstractwiki: Don't run our jobs too quickly, they'll collide and error (T430898)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:06 jforrester@deploy2003: Started scap sync-world: Backport for [[gerrit:1309112{{!}}abstractwiki: Don't run our jobs too quickly, they'll collide and error (T430898)]]
* 12:54 cgoubert@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 12:54 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:52 cgoubert@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 12:45 cgoubert@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 12:44 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 12:44 cgoubert@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 12:43 cgoubert@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:43 cgoubert@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:42 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org
* 12:20 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org
* 12:20 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org
* 12:18 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast4006.wikimedia.org
* 12:18 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:18 cmooney@cumin1003: END (PASS) - Cookbook sre.network.host-bgp (exit_code=0) for host ml-serve1004
* 12:18 cmooney@cumin1003: START - Cookbook sre.network.host-bgp for host ml-serve1004
* 12:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org
* 12:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org
* 12:10 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup[2003,2014].codfw.wmnet with reason: reboot &amp; upgrade
* 12:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet
* 12:09 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1003.eqiad.wmnet
* 12:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet
* 12:05 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid1003.eqiad.wmnet
* 12:03 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup[1003,1014].eqiad.wmnet with reason: reboot &amp; upgrade
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2003.codfw.wmnet
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 11:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid2003.codfw.wmnet
* 11:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 11:55 jmm@dns1004: END - running authdns-update
* 11:53 jmm@dns1004: START - running authdns-update
* 11:50 jmm@dns1004: END - running authdns-update
* 11:48 jmm@dns1004: START - running authdns-update
* 11:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org
* 11:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org
* 11:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet
* 11:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet
* 11:12 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet
* 11:12 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet
* 11:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org
* 11:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org
* 11:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org
* 11:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org
* 11:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 11:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2005.codfw.wmnet
* 11:00 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:00 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 10:59 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 10:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 10:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 10:55 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 10:51 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 10:51 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:50 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2005.codfw.wmnet
* 10:41 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:40 cmooney@cumin1003: END (PASS) - Cookbook sre.network.host-bgp (exit_code=0) for host ml-serve1003
* 10:40 cmooney@cumin1003: START - Cookbook sre.network.host-bgp for host ml-serve1003
* 10:39 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet
* 10:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org
* 10:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org
* 10:35 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1004.eqiad.wmnet with OS bookworm
* 10:31 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org
* 10:31 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org
* 10:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org
* 10:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org
* 10:29 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org
* 10:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org
* 10:23 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet
* 10:21 moritzm: failover Ganeti master in codfw/routed to ganeti2034 [[phab:T430928|T430928]]
* 10:19 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2031.codfw.wmnet to cluster codfw and group B
* 10:19 moritzm: readded ganeti2031 to the codfw Ganeti cluster [[phab:T430910|T430910]]
* 10:18 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1004.eqiad.wmnet with reason: host reimage
* 10:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org
* 10:18 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1003
* 10:18 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1003
* 10:17 jmm@cumin2003: START - Cookbook sre.ganeti.addnode for new host ganeti2031.codfw.wmnet to cluster codfw and group B
* 10:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org
* 10:16 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org
* 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org
* 10:15 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1004.eqiad.wmnet with reason: host reimage
* 10:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1001.eqiad.wmnet
* 10:14 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 10:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
* 10:00 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ml-serve1004
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1004
* 09:57 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
* 09:57 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1004
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1004.eqiad.wmnet 50.48.64.10.in-addr.arpa 0.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:57 klausman@cumin1003: START - Cookbook sre.dns.wipe-cache ml-serve1004.eqiad.wmnet 50.48.64.10.in-addr.arpa 0.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1004 - klausman@cumin1003&quot;
* 09:56 klausman@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1004 - klausman@cumin1003&quot;
* 09:55 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-coord1001.eqiad.wmnet
* 09:55 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 09:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet
* 09:52 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-coord1001.eqiad.wmnet
* 09:52 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 09:52 klausman@cumin1003: START - Cookbook sre.dns.netbox
* 09:50 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
* 09:50 klausman@cumin1003: START - Cookbook sre.hosts.move-vlan for host ml-serve1004
* 09:50 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1004.eqiad.wmnet with OS bookworm
* 09:50 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1003.eqiad.wmnet with OS bookworm
* 09:50 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1001.eqiad.wmnet
* 09:49 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 09:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2007.codfw.wmnet
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 09:49 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet
* 09:49 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 09:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet
* 09:39 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet
* 09:39 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 09:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet
* 09:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet
* 09:35 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet
* 09:34 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2007.codfw.wmnet
* 09:33 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet
* 09:33 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet
* 09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet
* 09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:27 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 09:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet
* 09:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet
* 09:23 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309102{{!}}NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] (duration: 06m 27s)
* 09:23 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2205: codfw rack B4 repool after maintenance
* 09:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:23 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2204: codfw rack B4 repool after maintenance
* 09:19 urbanecm@deploy2003: urbanecm: Continuing with deployment
* 09:19 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1309102{{!}}NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:18 jmm@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: reboot
* 09:17 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1309102{{!}}NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]]
* 09:08 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ml-serve1003
* 09:08 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1003
* 09:07 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1003
* 09:07 klausman@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1003.eqiad.wmnet 81.32.64.10.in-addr.arpa 1.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:07 klausman@cumin1003: START - Cookbook sre.dns.wipe-cache ml-serve1003.eqiad.wmnet 81.32.64.10.in-addr.arpa 1.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:07 klausman@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:07 klausman@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1003 - klausman@cumin1003&quot;
* 09:06 klausman@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1003 - klausman@cumin1003&quot;
* 08:58 klausman@cumin1003: START - Cookbook sre.dns.netbox
* 08:57 klausman@cumin1003: START - Cookbook sre.hosts.move-vlan for host ml-serve1003
* 08:57 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1003.eqiad.wmnet with OS bookworm
* 08:55 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve1003.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 08:55 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet
* 08:55 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet
* 08:55 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1003.eqiad.wmnet with OS bookworm
* 08:39 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 08:38 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2205: codfw rack B4 repool after maintenance
* 08:37 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2204: codfw rack B4 repool after maintenance
* 08:36 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 08:35 hashar@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 08:32 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:32 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:31 hashar@deploy2003: Rolling back deployment
* 08:26 moritzm: failover Ganeti master in codfw to ganeti2048 [[phab:T430928|T430928]]
* 08:16 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1003.eqiad.wmnet with OS bookworm
* 08:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2004.codfw.wmnet
* 08:16 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet
* 08:16 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet
* 08:16 klausman@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve1003.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 08:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet
* 08:15 XioNoX: lsw1-b4-codfw&gt; request system reboot - [[phab:T430910|T430910]]
* 08:15 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b4-codfw,lsw1-b4-codfw IPv6,lsw1-b4-codfw.mgmt with reason: Switch maintenance
* 08:13 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for codfw rack B4
* 08:13 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2004.codfw.wmnet
* 08:10 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet
* 08:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2205: codfw rack B4 depool for maintenance
* 08:08 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2205: codfw rack B4 depool for maintenance
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2003.codfw.wmnet
* 08:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2204: codfw rack B4 depool for maintenance
* 08:08 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2204: codfw rack B4 depool for maintenance
* 08:08 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 27 hosts with reason: codfw rack B4 depool for maintenance
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cumin2003.codfw.wmnet
* 07:56 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack B4
* 07:54 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1008.eqiad.wmnet with OS trixie
* 07:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308687{{!}}Enable sub-references on more group2 wikis (batch2) (T430941)]] (duration: 08m 36s)
* 07:44 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 07:43 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1308687{{!}}Enable sub-references on more group2 wikis (batch2) (T430941)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:41 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1308687{{!}}Enable sub-references on more group2 wikis (batch2) (T430941)]]
* 07:35 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1008.eqiad.wmnet with reason: host reimage
* 07:31 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1008.eqiad.wmnet with reason: host reimage
* 07:12 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1008.eqiad.wmnet with OS trixie
* 07:10 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 07:00 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 06:59 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1008.eqiad.wmnet with OS trixie
* 06:57 Emperor: rebalance thanos swift rings after previous re-image of thanos-fe1004 to trixie
* 06:47 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1008.eqiad.wmnet with OS trixie
* 04:10 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cp6008.drmrs.wmnet with reason: Hardware failure - [[phab:T431651|T431651]]
* 03:55 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6008.*
* 03:29 ryankemper: [[phab:T431311|T431311]] Repooled eqiad cirrussearch clusters (`chi/omega/psi`) following completion of OpenSearch 2.19 migration
* 03:27 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=eqiad
* 03:27 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=eqiad
* 03:27 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-08 ==
* 23:52 Amir1: ladsgroup@deploy2003:~$ mwscript-k8s --follow -- extensions/ORES/maintenance/PurgeScoreCache.php --wiki=simplewiki --model damaging --old ([[phab:T431159|T431159]])
* 23:46 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 23:46 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add missing PTR for 2001:df2:e500:fe08::1 - cmooney@cumin1003&quot;
* 23:46 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add missing PTR for 2001:df2:e500:fe08::1 - cmooney@cumin1003&quot;
* 23:40 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 23:16 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 23:15 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 22:42 rzl@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 22:40 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308782{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] (duration: 12m 55s)
* 22:40 rzl@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 22:37 rzl@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 22:36 rzl@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 22:35 rzl@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 22:34 urbanecm@deploy2003: urbanecm: Continuing with deployment
* 22:33 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1308782{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 rzl@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 22:32 rzl@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 22:30 rzl@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 22:30 rzl@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 22:29 rzl@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 22:27 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1308782{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]]
* 22:26 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 22:22 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 22:21 rzl@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 22:19 rzl@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 22:19 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 22:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 22:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 22:13 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 22:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 22:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 22:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 22:06 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 22:05 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1094.eqiad.wmnet with OS trixie
* 22:01 urbanecm: Make https://test.wikipedia.org/w/index.php?title=MediaWiki:GrowthExperimentsSuggestedEdits.json&amp;diff=prev&amp;oldid=750552 with GrowthExperiments disabled (via mw-experimental), then run `\MediaWiki\MediaWikiServices::getInstance()-&gt;get('CommunityConfiguration.ProviderFactory')-&gt;newProvider('GrowthSuggestedEdits')-&gt;getStore()-&gt;invalidate()` ([[phab:T431625|T431625]])
* 21:56 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d2-codfw
* 21:55 urbanecm@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 21:55 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d2-codfw
* 21:55 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c4-codfw
* 21:55 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c4-codfw
* 21:55 urbanecm@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2002
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2002
* 21:54 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2002
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2002.codfw.wmnet 50.32.192.10.in-addr.arpa 0.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:54 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2002.codfw.wmnet 50.32.192.10.in-addr.arpa 0.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2002 - bking@cumin2003&quot;
* 21:54 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2002 - bking@cumin2003&quot;
* 21:49 bking@cumin2003: START - Cookbook sre.dns.netbox
* 21:49 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2002
* 21:49 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2002.codfw.wmnet with OS trixie
* 21:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1094.eqiad.wmnet with reason: host reimage
* 21:42 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 21:39 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 21:37 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1094.eqiad.wmnet with reason: host reimage
* 21:36 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 21:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 21:29 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 21:27 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 21:22 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1094.eqiad.wmnet with OS trixie
* 21:21 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2039.codfw.wmnet with OS bullseye
* 21:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 21:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 21:04 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2039.codfw.wmnet with reason: host reimage
* 21:00 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2039.codfw.wmnet with reason: host reimage
* 20:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1073.eqiad.wmnet with OS trixie
* 20:48 mutante: deploy2003 - kill 1102 (stunnel4) ; systemctl start stunnel4 ([[phab:T418262|T418262]])
* 20:42 cjming@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308183{{!}}Move Test Kitchen config from CommonSettings.php (T431257)]] (duration: 33m 02s)
* 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2039.codfw.wmnet with OS bullseye
* 20:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1073.eqiad.wmnet with reason: host reimage
* 20:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1073.eqiad.wmnet with reason: host reimage
* 20:30 cjming@deploy2003: cjming: Continuing with deployment
* 20:28 cjming@deploy2003: cjming: Backport for [[gerrit:1308183{{!}}Move Test Kitchen config from CommonSettings.php (T431257)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:20 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1098.eqiad.wmnet with OS trixie
* 20:13 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1073.eqiad.wmnet with OS trixie
* 20:09 cjming@deploy2003: Started scap sync-world: Backport for [[gerrit:1308183{{!}}Move Test Kitchen config from CommonSettings.php (T431257)]]
* 20:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1098.eqiad.wmnet with reason: host reimage
* 19:56 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1098.eqiad.wmnet with reason: host reimage
* 19:55 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d4-codfw
* 19:54 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d4-codfw
* 19:54 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-codfw
* 19:54 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c1-codfw
* 19:52 mutante: restarting gerrit on gerrit.wikimedia.org (gerrit2003)
* 19:48 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2331.codfw.wmnet
* 19:48 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2331.codfw.wmnet
* 19:48 mutante: restarting gerrit on gerrit-replica.wikimedia.org (gerrit1003)
* 19:46 mutante: restarting gerrit on gerrit-spare.wikimedia.org (gerrit2002)
* 19:43 jasmine@cumin2002: conftool action : set/pooled=yes; selector: name=wikikube-worker2331.codfw.wmnet,cluster=kubernetes,service=kubesvc
* 19:43 jasmine@cumin2002: conftool action : set/weight=10; selector: name=wikikube-worker2331.codfw.wmnet,cluster=kubernetes,service=kubesvc
* 19:40 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1098.eqiad.wmnet with OS trixie
* 19:38 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d5-codfw
* 19:38 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d5-codfw
* 19:38 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c7-codfw
* 19:38 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c7-codfw
* 19:38 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c5-codfw
* 19:38 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c5-codfw
* 19:30 jasmine_: ran homer on lsw1-d8-codfw, adding wikikube-worker2331 to cluster
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1100.eqiad.wmnet with OS trixie
* 19:20 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d8-codfw
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d8-codfw
* 19:19 mutante: gerrit - replacing private key for registerEmail verification
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-magru
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device cr2-magru
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d7-codfw
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d7-codfw
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d3-codfw
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d1-codfw
* 19:18 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d1-codfw
* 19:18 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c2-codfw
* 19:18 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c2-codfw
* 19:11 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-codfw
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d1-codfw
* 19:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-magru
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device cr1-magru
* 19:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d8-codfw
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d8-codfw
* 19:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-codfw
* 19:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1100.eqiad.wmnet with reason: host reimage
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d6-codfw
* 19:09 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c6-codfw
* 19:09 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c6-codfw
* 19:09 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c3-codfw
* 19:09 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c3-codfw
* 19:09 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b4-magru
* 19:09 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-b4-magru
* 19:08 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b3-magru
* 19:08 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-b3-magru
* 19:05 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1100.eqiad.wmnet with reason: host reimage
* 19:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1122.eqiad.wmnet with OS trixie
* 19:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 18:59 topranks: rolling out update to BGP ACL on Nokia Switches eqiad, codfw &amp; ulsfo [[phab:T425703|T425703]]
* 18:58 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 18:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 18:55 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 18:53 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 18:52 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 18:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1100.eqiad.wmnet with OS trixie
* 18:48 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1068.eqiad.wmnet with OS trixie
* 18:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1102.eqiad.wmnet with OS trixie
* 18:47 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 18:46 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 18:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1122.eqiad.wmnet with reason: host reimage
* 18:38 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1122.eqiad.wmnet with reason: host reimage
* 18:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1068.eqiad.wmnet with reason: host reimage
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1122
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1122
* 18:25 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1122
* 18:25 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1122.eqiad.wmnet 31.48.64.10.in-addr.arpa 1.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:25 bking@cumin2003: START - Cookbook sre.dns.wipe-cache cirrussearch1122.eqiad.wmnet 31.48.64.10.in-addr.arpa 1.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:25 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:25 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1122 - bking@cumin2003&quot;
* 18:25 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1122 - bking@cumin2003&quot;
* 18:21 rzl@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 18:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1068.eqiad.wmnet with reason: host reimage
* 18:21 rzl@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 18:21 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 18:20 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 18:19 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 18:19 bking@cumin2003: START - Cookbook sre.dns.netbox
* 18:18 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1122
* 18:18 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1122.eqiad.wmnet with OS trixie
* 18:15 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 18:15 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 18:13 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 18:13 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 18:10 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 18:05 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1068.eqiad.wmnet with OS trixie
* 18:01 kamila@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - [[phab:T423714|T423714]] (duration: 18m 29s)
* 18:00 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:55 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:46 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:42 kamila@deploy2003: Started scap sync-world: Test deployment to validate deployment server switchover - [[phab:T423714|T423714]]
* 17:42 kamila@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - [[phab:T423714|T423714]] (duration: 19m 50s)
* 17:42 kamila@deploy2003: Rolling back deployment
* 17:35 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:31 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 17:18 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 17:16 kamila@deploy1003: Unlocked for deployment [MediaWiki]: switching deployment server (duration: 22m 07s)
* 17:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 17:11 kamila@dns1005: END - running authdns-update
* 17:09 kamila@dns1005: START - running authdns-update
* 17:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 17:04 jasmine@cumin2002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1164.eqiad.wmnet
* 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1164.eqiad.wmnet
* 17:04 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1164.eqiad.wmnet
* 16:56 kamila@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet,releases1003.eqiad.wmnet with reason: Deployment server switchover
* 16:54 kamila@deploy1003: Locking from deployment [MediaWiki]: switching deployment server
* 16:53 kamila@deploy1003: Unlocked for deployment [MediaWiki]: switching deployment server (duration: 04m 02s)
* 16:52 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie
* 16:49 kamila@deploy1003: Locking from deployment [MediaWiki]: switching deployment server
* 16:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1095.eqiad.wmnet with OS trixie
* 16:45 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1093.eqiad.wmnet with OS trixie
* 16:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1164.eqiad.wmnet with OS trixie
* 16:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1095.eqiad.wmnet with reason: host reimage
* 16:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 16:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 16:23 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage
* 16:18 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cirrussearch1093.eqiad.wmnet with reason: host reimage
* 16:16 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage
* 16:15 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1095.eqiad.wmnet with reason: host reimage
* 16:09 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 16:09 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 16:08 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1093.eqiad.wmnet with reason: host reimage
* 15:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Pool test
* 15:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:59 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Pool test
* 15:58 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Depool test
* 15:58 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:58 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:58 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Depool test
* 15:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1164
* 15:57 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1164
* 15:57 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1164
* 15:56 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1164.eqiad.wmnet 114.48.64.10.in-addr.arpa 4.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:56 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1164.eqiad.wmnet 114.48.64.10.in-addr.arpa 4.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:56 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:56 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1164 - jasmine@cumin2002&quot;
* 15:56 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1164 - jasmine@cumin2002&quot;
* 15:52 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1093.eqiad.wmnet with OS trixie
* 15:51 jasmine@cumin2002: START - Cookbook sre.dns.netbox
* 15:51 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1164
* 15:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1164.eqiad.wmnet with OS trixie
* 15:50 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet
* 15:50 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet
* 15:50 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1164.eqiad.wmnet
* 15:48 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1095.eqiad.wmnet with OS trixie
* 15:42 jasmine@cumin2002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1164.eqiad.wmnet
* 15:42 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet
* 15:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:42 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet
* 15:42 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1164.eqiad.wmnet
* 15:39 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 15:39 elukey@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 15:36 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1007.eqiad.wmnet with OS trixie
* 15:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Pool test
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:19 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage
* 15:15 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
* 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet
* 15:15 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
* 15:15 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 15:15 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage
* 15:15 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:14 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Pool test
* 15:14 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet
* 15:14 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: Depool test
* 15:14 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2228: Depool test
* 15:10 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 15:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 15:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 15:08 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 15:06 blake@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet
* 15:06 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 15:06 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 15:05 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet
* 15:05 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 15:04 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:04 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 15:04 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:03 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 15:03 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 15:03 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:03 swfrench-wmf: restarted navtiming on webperf2003 - [[phab:T430909|T430909]]
* 15:03 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:03 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 15:03 swfrench-wmf: restarted eqsin, codfw confds - [[phab:T430909|T430909]]
* 15:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test1002.eqiad.wmnet
* 15:02 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet
* 14:59 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:59 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:55 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS trixie
* 14:52 swfrench-wmf: restarted ulsfo confds, confirmed now connected to codfw backends except those using wikimedia.org SRV record - [[phab:T430909|T430909]]
* 14:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:50 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:49 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 14:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:41 moritzm: uninstalling dhcpcd-base from trixie hosts which still have it installed [[phab:T414341|T414341]]
* 14:40 sukhe: sudo cumin -b1 -s120 &quot;P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2011*&lt;nowiki&gt;}&lt;/nowiki&gt; or P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2012*&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; &quot;systemctl restart pybal.service&quot;
* 14:39 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 14:39 mvernon@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host thanos-be1007.eqiad.wmnet with OS trixie
* 14:37 sukhe: restart pybal on lvs2013 to revert back to conf2004
* 14:35 sukhe: restart pybal on lvs2014 to revert back to conf2004
* 14:34 swfrench-wmf: switched codfw, eqsin, ulsfo etcd client SRV records back to codfw - [[phab:T430909|T430909]]
* 14:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1002.eqiad.wmnet
* 14:32 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet
* 14:31 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS trixie
* 14:31 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:31 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:31 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:30 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:30 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Pool test
* 14:30 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:29 swfrench@dns1004: END - running authdns-update
* 14:29 moritzm: installing jackson-core security updates
* 14:27 swfrench@dns1004: START - running authdns-update
* 14:22 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:22 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:22 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1119.eqiad.wmnet with OS trixie
* 14:22 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:21 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:20 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:20 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 14:20 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:19 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 14:19 moritzm: installing librabbitmq security updates
* 14:19 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1002.eqiad.wmnet
* 14:18 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet
* 14:16 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:16 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:16 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:15 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:15 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:14 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Pool test
* 14:14 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 14:14 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet
* 14:10 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 14:08 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet
* 14:05 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1118.eqiad.wmnet with OS trixie
* 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test1001.eqiad.wmnet
* 14:00 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-worker@eqiad
* 14:00 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 13:59 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 13:57 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1119.eqiad.wmnet with reason: host reimage
* 13:54 moritzm: installing libcap2 security updates
* 13:53 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1119.eqiad.wmnet with reason: host reimage
* 13:52 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet
* 13:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0)
* 13:52 fceratto@cumin1003: START - Cookbook sre.mysql.depool
* 13:50 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-worker@eqiad
* 13:50 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1051.eqiad.wmnet
* 13:50 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet
* 13:50 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet
* 13:49 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 13:45 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1081.eqiad.wmnet with OS trixie
* 13:41 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1119
* 13:41 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1119
* 13:40 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1119
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1119.eqiad.wmnet 97.32.64.10.in-addr.arpa 7.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:40 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1119.eqiad.wmnet 97.32.64.10.in-addr.arpa 7.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1119 - atsuko@cumin1003&quot;
* 13:40 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1119 - atsuko@cumin1003&quot;
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1118.eqiad.wmnet with reason: host reimage
* 13:39 moritzm: installing krb5 security updates
* 13:37 Lucas_WMDE: UTC afternoon backport+config window done
* 13:37 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1006.eqiad.wmnet with OS trixie
* 13:36 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1118.eqiad.wmnet with reason: host reimage
* 13:36 atsuko@cumin1003: START - Cookbook sre.dns.netbox
* 13:35 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308622{{!}}Enable Article Guidance extension on itwiki (T431540)]] (duration: 07m 46s)
* 13:34 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1119
* 13:34 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1119.eqiad.wmnet with OS trixie
* 13:30 sbisson@deploy1003: sbisson: Continuing with deployment
* 13:30 moritzm: installing openssh security updates
* 13:30 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
* 13:29 sbisson@deploy1003: sbisson: Backport for [[gerrit:1308622{{!}}Enable Article Guidance extension on itwiki (T431540)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 13:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 13:27 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1308622{{!}}Enable Article Guidance extension on itwiki (T431540)]]
* 13:26 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1051.eqiad.wmnet with OS trixie
* 13:24 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage
* 13:24 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1118
* 13:24 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1118
* 13:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308553{{!}}Deploy IRS to enwiki (T431316)]] (duration: 12m 12s)
* 13:21 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage
* 13:18 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage
* 13:18 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1118
* 13:18 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1118.eqiad.wmnet 90.32.64.10.in-addr.arpa 0.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:18 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1118.eqiad.wmnet 90.32.64.10.in-addr.arpa 0.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:18 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:18 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1118 - atsuko@cumin1003&quot;
* 13:18 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1118 - atsuko@cumin1003&quot;
* 13:17 stran@deploy1003: stran: Continuing with deployment
* 13:16 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:15 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:13 atsuko@cumin1003: START - Cookbook sre.dns.netbox
* 13:12 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1118
* 13:12 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage
* 13:12 stran@deploy1003: stran: Backport for [[gerrit:1308553{{!}}Deploy IRS to enwiki (T431316)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1118.eqiad.wmnet with OS trixie
* 13:10 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1308553{{!}}Deploy IRS to enwiki (T431316)]]
* 13:05 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-worker@codfw
* 13:05 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 13:05 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1081.eqiad.wmnet with OS trixie
* 13:05 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage
* 13:04 moritzm: installing jq security updates
* 13:04 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 13:01 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage
* 12:58 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-worker@codfw
* 12:52 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:50 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1051
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1051
* 12:43 moritzm: installing Python 3.11 security updates
* 12:43 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1051
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1051.eqiad.wmnet 46.32.64.10.in-addr.arpa 6.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:43 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1051.eqiad.wmnet 46.32.64.10.in-addr.arpa 6.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1051 - blake@cumin1003&quot;
* 12:43 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1051 - blake@cumin1003&quot;
* 12:38 blake@cumin1003: START - Cookbook sre.dns.netbox
* 12:38 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1051
* 12:38 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1051.eqiad.wmnet with OS trixie
* 12:37 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet
* 12:36 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet
* 12:36 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1051.eqiad.wmnet
* 12:34 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:34 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:27 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:27 mvernon@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 12:02 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:01 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1006.eqiad.wmnet with OS trixie
* 11:43 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 11:38 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 11:32 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1076.eqiad.wmnet with OS trixie
* 11:26 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1075.eqiad.wmnet with OS trixie
* 11:21 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet
* 11:19 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet
* 11:19 moritzm: temporarily remove ganeti2031 from codfw cluster [[phab:T430910|T430910]]
* 11:08 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1076.eqiad.wmnet with reason: host reimage
* 11:08 moritzm: installing Linux 6.1.176 on Bookworm servers
* 11:03 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1076.eqiad.wmnet with reason: host reimage
* 11:00 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1075.eqiad.wmnet with reason: host reimage
* 10:56 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1075.eqiad.wmnet with reason: host reimage
* 10:47 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1076.eqiad.wmnet with OS trixie
* 10:46 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1074.eqiad.wmnet with OS trixie
* 10:45 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1005.eqiad.wmnet with OS trixie
* 10:40 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1075.eqiad.wmnet with OS trixie
* 10:32 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 10:29 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage
* 10:25 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage
* 10:24 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1074.eqiad.wmnet with reason: host reimage
* 10:17 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1074.eqiad.wmnet with reason: host reimage
* 10:12 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1005.eqiad.wmnet with OS trixie
* 10:12 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet
* 10:04 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 10:01 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet
* 10:01 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1074.eqiad.wmnet with OS trixie
* 10:01 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 09:43 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 09:43 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
* 09:43 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 09:35 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
* 09:34 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 09:34 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 09:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 41 days, 15:00:00 on db2252.codfw.wmnet with reason: Test
* 09:32 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 09:32 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 09:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: codfw rack B3 pool after maintenance
* 09:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 09:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 09:07 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 09:02 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0)
* 08:57 topranks: merge patch to shift eqiad &lt;-&gt; esams traffic onto new 40G circuit
* 08:54 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 08:50 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1004.eqiad.wmnet with OS trixie
* 08:50 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart
* 08:50 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99)
* 08:50 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart
* 08:45 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool es2051: codfw rack B3 pool after maintenance
* 08:44 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:44 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:43 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet
* 08:43 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet
* 08:42 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 08:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 08:40 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 08:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:35 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis minwikiquote in section s3
* 08:33 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s3
* 08:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5
* 08:30 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5
* 08:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis minwikiquote in section s5
* 08:27 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage
* 08:23 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage
* 08:23 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:22 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s5
* 08:19 XioNoX: lsw1-b3-codfw&gt; request system reboot - [[phab:T430909|T430909]]
* 08:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5
* 08:17 hashar@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 08:16 cwilliams@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5
* 08:15 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for codfw rack B3
* 08:15 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet
* 08:15 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-b3-codfw,lsw1-b3-codfw IPv6,lsw1-b3-codfw.mgmt with reason: Switch maintenance
* 08:15 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet
* 08:13 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: codfw rack B3 depool for maintenance
* 08:05 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool es2051: codfw rack B3 depool for maintenance
* 08:05 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1004.eqiad.wmnet with OS trixie
* 08:03 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1005.eqiad.wmnet with OS trixie
* 08:03 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 21 hosts with reason: codfw rack B3 depool for maintenance
* 07:56 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack B3
* 07:44 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1005.eqiad.wmnet with reason: host reimage
* 07:38 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1005.eqiad.wmnet with reason: host reimage
* 07:36 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet with OS bookworm
* 07:29 moritzm: installing gnutls28 security updates
* 07:21 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1005.eqiad.wmnet with OS trixie
* 07:13 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1125.eqiad.wmnet with OS trixie
* 07:13 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1005.eqiad.wmnet with reason: host reimage
* 07:07 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1005.eqiad.wmnet with reason: host reimage
* 06:56 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1005.eqiad.wmnet with OS bookworm
* 06:54 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1125.eqiad.wmnet with reason: host reimage
* 06:52 elukey: upgrade all trixie hosts to pywmflib 3.1 - [[phab:T430552|T430552]]
* 06:50 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1125.eqiad.wmnet with reason: host reimage
* 06:40 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 06:40 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 06:38 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1125.eqiad.wmnet with OS trixie
* 05:42 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1107.eqiad.wmnet with OS trixie
* 05:35 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1124.eqiad.wmnet with OS trixie
* 05:31 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1101.eqiad.wmnet with OS trixie
* 05:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1107.eqiad.wmnet with reason: host reimage
* 05:17 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1124.eqiad.wmnet with reason: host reimage
* 05:13 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1107.eqiad.wmnet with reason: host reimage
* 05:13 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1101.eqiad.wmnet with reason: host reimage
* 05:11 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1124.eqiad.wmnet with reason: host reimage
* 05:10 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1101.eqiad.wmnet with reason: host reimage
* 04:58 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1124.eqiad.wmnet with OS trixie
* 04:56 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1107.eqiad.wmnet with OS trixie
* 04:55 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1101.eqiad.wmnet with OS trixie
* 02:27 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308251{{!}}Activate minwikiquote (T429922)]] (duration: 08m 14s)
* 02:22 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 02:21 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1308251{{!}}Activate minwikiquote (T429922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 02:19 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1308251{{!}}Activate minwikiquote (T429922)]]
* 01:59 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308250{{!}}Init minwikiquote (T429922)]] (duration: 09m 46s)
* 01:55 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 01:51 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1308250{{!}}Init minwikiquote (T429922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 01:49 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1308250{{!}}Init minwikiquote (T429922)]]
* 01:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1099.eqiad.wmnet with OS trixie
* 00:57 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1110.eqiad.wmnet with OS trixie
* 00:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1099.eqiad.wmnet with reason: host reimage
* 00:41 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1099.eqiad.wmnet with reason: host reimage
* 00:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1110.eqiad.wmnet with reason: host reimage
* 00:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1110.eqiad.wmnet with reason: host reimage
* 00:26 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1099.eqiad.wmnet with OS trixie
* 00:15 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1110.eqiad.wmnet with OS trixie

== 2026-07-07 ==
* 22:49 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1097.eqiad.wmnet with OS trixie
* 22:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1097.eqiad.wmnet with reason: host reimage
* 22:24 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1097.eqiad.wmnet with reason: host reimage
* 22:14 hashar: Restarting Gerrit on gerrit2002 and gerrit1003 (replicas)
* 22:09 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1097.eqiad.wmnet with OS trixie
* 22:07 hashar: Restarting Gerrit on gerrit2003
* 21:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:11 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:05 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1108.eqiad.wmnet with OS trixie
* 20:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1091.eqiad.wmnet with OS trixie
* 20:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1090.eqiad.wmnet with OS trixie
* 20:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1108.eqiad.wmnet with reason: host reimage
* 20:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1091.eqiad.wmnet with reason: host reimage
* 20:33 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1090.eqiad.wmnet with reason: host reimage
* 20:33 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1091.eqiad.wmnet with reason: host reimage
* 20:30 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1108.eqiad.wmnet with reason: host reimage
* 20:30 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 20:30 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1090.eqiad.wmnet with reason: host reimage
* 20:30 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 20:27 jasmine_: &quot;homer lsw1-c2-eqiad* commit &quot;Added new stacked control plane wikikube-ctrl1006&quot;&quot;
* 20:22 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308196{{!}}Bump Parsoid image limit to 1250 (T430854)]] (duration: 07m 29s)
* 20:20 jasmine_: &quot;homer &quot;cr*eqiad*&quot; commit &quot;Added new stacked control plane wikikube-ctrl1006&quot;&quot;
* 20:18 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1091.eqiad.wmnet with OS trixie
* 20:17 arlolra@deploy1003: arlolra: Continuing with deployment
* 20:16 arlolra@deploy1003: arlolra: Backport for [[gerrit:1308196{{!}}Bump Parsoid image limit to 1250 (T430854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:16 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1090.eqiad.wmnet with OS trixie
* 20:15 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1108.eqiad.wmnet with OS trixie
* 20:14 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1308196{{!}}Bump Parsoid image limit to 1250 (T430854)]]
* 20:09 cwhite: remove 2026-04 swift log archives from centrallog2002 to free some space
* 20:01 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cirrussearch1108.eqiad.wmnet with OS trixie
* 19:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1108.eqiad.wmnet with OS trixie
* 19:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1090.eqiad.wmnet with OS trixie
* 19:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1109.eqiad.wmnet with OS trixie
* 19:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1092.eqiad.wmnet with OS trixie
* 19:32 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1123.eqiad.wmnet with OS trixie
* 19:22 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1109.eqiad.wmnet with reason: host reimage
* 19:19 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=dns7002.*
* 19:17 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1092.eqiad.wmnet with reason: host reimage
* 19:17 jasmine@dns1004: END - running authdns-update
* 19:15 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1109.eqiad.wmnet with reason: host reimage
* 19:15 jasmine@dns1004: START - running authdns-update
* 19:15 cdobbins@dns1004: END - running authdns-update
* 19:14 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1123.eqiad.wmnet with reason: host reimage
* 19:13 cdobbins@dns1004: START - running authdns-update
* 19:12 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=dns7002.*,service=authdns-update
* 19:11 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1092.eqiad.wmnet with reason: host reimage
* 19:10 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1123.eqiad.wmnet with reason: host reimage
* 18:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1123.eqiad.wmnet with OS trixie
* 18:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1109.eqiad.wmnet with OS trixie
* 18:56 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1092.eqiad.wmnet with OS trixie
* 18:52 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 18:49 swfrench@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 18:40 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 18:38 swfrench@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 18:11 swfrench-wmf: restarted eqsin, codfw confds - [[phab:T430909|T430909]]
* 18:01 swfrench-wmf: restarted navtiming on webperf2003 - [[phab:T430909|T430909]]
* 17:59 swfrench-wmf: restarted ulsfo confds, confirmed now connected to eqiad backends - [[phab:T430909|T430909]]
* 17:52 sukhe: restart pybal on lvs2011 to switch from conf2004 to conf1008: [[phab:T430909|T430909]]
* 17:51 sukhe: restart pybal on lvs2012 to switch from conf2004 to conf1008 [puppet re-enabled there]: [[phab:T430909|T430909]]
* 17:46 sukhe: restart pybal on lvs2013 to switch from conf2004 to conf1008: [[phab:T430909|T430909]]
* 17:44 swfrench-wmf: switched codfw, eqsin, ulsfo etcd client SRV records to eqiad - [[phab:T430909|T430909]]
* 17:43 swfrench@dns1004: END - running authdns-update
* 17:40 swfrench@dns1004: START - running authdns-update
* 17:40 sukhe: restart pybal on lvs2014 to switch from conf2004 to conf1008: [[phab:T430909|T430909]]
* 17:21 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet
* 17:15 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet
* 17:14 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet
* 17:06 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet
* 17:06 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service'  # lvs2013, [[phab:T416623|T416623]]
* 17:04 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet
* 17:02 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1111.eqiad.wmnet with OS trixie
* 17:00 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service'  # lvs2014, [[phab:T416623|T416623]]
* 16:58 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet
* 16:58 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS bookworm
* 16:55 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service'  # lvs1019, [[phab:T416623|T416623]]
* 16:53 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-eqiad' 'systemctl restart pybal.service'  # lvs1020, [[phab:T416623|T416623]]
* 16:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1111.eqiad.wmnet with reason: host reimage
* 16:40 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1111.eqiad.wmnet with reason: host reimage
* 16:38 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 47794
* 16:35 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 47794
* 16:27 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1111.eqiad.wmnet with OS trixie
* 16:21 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1006.eqiad.wmnet with OS trixie
* 16:06 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 16:03 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1006.eqiad.wmnet with reason: host reimage
* 15:58 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1121.eqiad.wmnet with OS trixie
* 15:58 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 15:56 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1006.eqiad.wmnet with reason: host reimage
* 15:54 mutante: jenkins down in planned maintenance window
* 15:42 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1037.eqiad.wmnet
* 15:42 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet
* 15:42 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet
* 15:41 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1006.eqiad.wmnet with OS trixie
* 15:34 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1121.eqiad.wmnet with reason: host reimage
* 15:33 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm
* 15:33 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS trixie
* 15:30 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1121.eqiad.wmnet with reason: host reimage
* 15:29 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org
* 15:20 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org
* 15:18 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1121
* 15:18 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1121
* 15:18 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1002.wikimedia.org
* 15:17 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1121
* 15:17 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:17 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 15:16 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
* 15:16 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:16 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1121 - atsuko@cumin1003&quot;
* 15:16 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1121 - atsuko@cumin1003&quot;
* 15:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1037.eqiad.wmnet with OS trixie
* 15:11 atsuko@cumin1003: START - Cookbook sre.dns.netbox
* 15:09 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
* 15:09 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1121
* 15:09 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host clouddumps1002.wikimedia.org
* 15:09 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
* 15:09 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1121.eqiad.wmnet with OS trixie
* 15:08 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1007.eqiad.wmnet with OS trixie
* 15:08 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host clouddumps1002.wikimedia.org
* 15:08 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
* 15:05 brennen@deploy1003: Finished deploy [phabricator/deployment@7e02037]: deploy phab1004 for [[phab:T431440|T431440]] (duration: 00m 47s)
* 15:04 brennen@deploy1003: Started deploy [phabricator/deployment@7e02037]: deploy phab1004 for [[phab:T431440|T431440]]
* 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@7e02037]: deploy phab2003 for [[phab:T431440|T431440]] (duration: 00m 51s)
* 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@7e02037]: deploy phab2003 for [[phab:T431440|T431440]]
* 15:00 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71] (thin): Regular analytics weekly train THIN [analytics/refinery@7d8dc71f] (duration: 02m 10s)
* 14:58 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71] (thin): Regular analytics weekly train THIN [analytics/refinery@7d8dc71f]
* 14:58 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71]: Regular analytics weekly train [analytics/refinery@7d8dc71f] (duration: 04m 14s)
* 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage
* 14:53 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71]: Regular analytics weekly train [analytics/refinery@7d8dc71f]
* 14:53 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7d8dc71f] (duration: 02m 00s)
* 14:51 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7d8dc71f]
* 14:51 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab2003.codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: maintenance
* 14:51 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage
* 14:50 JavierMonton: Deploying Refinery at {{Gerrit|7d8dc71f}} for change {{Gerrit|1308087}} / [[phab:T431318|T431318]] - update filerevision table sqoop and table
* 14:49 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1007.eqiad.wmnet with reason: host reimage
* 14:42 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1007.eqiad.wmnet with reason: host reimage
* 14:40 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1083.eqiad.wmnet with OS trixie
* 14:37 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:36 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:35 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-master@eqiad
* 14:35 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 14:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1037
* 14:34 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1037
* 14:34 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:cleanMentorList.php --wiki=frwiki  # [[phab:T427386|T427386]]
* 14:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 14:34 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308112{{!}}Revert^2 &quot;[Growth] frwiki: Deploy automated mentor list cleaner&quot; (T427386)]] (duration: 06m 47s)
* 14:34 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 14:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:33 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:32 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1037
* 14:31 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:31 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:29 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-master@eqiad
* 14:29 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:29 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:29 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:28 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:27 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:27 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1308112{{!}}Revert^2 &quot;[Growth] frwiki: Deploy automated mentor list cleaner&quot; (T427386)]]
* 14:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:26 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1007.eqiad.wmnet with OS trixie
* 14:26 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:26 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
* 14:26 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:25 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:25 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:25 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:cleanMentorList.php --wiki=frwiki  # [[phab:T427386|T427386]]
* 14:24 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:24 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1037 - blake@cumin1003&quot;
* 14:24 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1037 - blake@cumin1003&quot;
* 14:20 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1083.eqiad.wmnet with reason: host reimage
* 14:19 blake@cumin1003: START - Cookbook sre.dns.netbox
* 14:19 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-master@codfw
* 14:19 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 14:19 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1037
* 14:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1037.eqiad.wmnet with OS trixie
* 14:18 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet
* 14:18 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 14:18 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet
* 14:18 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1037.eqiad.wmnet
* 14:16 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2007.codfw.wmnet with OS trixie
* 14:16 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1083.eqiad.wmnet with reason: host reimage
* 14:15 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1036.eqiad.wmnet
* 14:15 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet
* 14:14 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet
* 14:12 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-master@codfw
* 14:11 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1120.eqiad.wmnet with OS trixie
* 14:05 moritzm: installing distro-info-data updates from trixie/bookworm point releases
* 14:04 fabfur: disable puppet on A:cp-text to selectively apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308040
* 14:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308106{{!}}fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105{{!}}fix(MentorListCleaner): Do not access property before inicialization (T430689)]] (duration: 27m 48s)
* 14:00 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 14:00 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1083.eqiad.wmnet with OS trixie
* 13:58 urbanecm@deploy1003: urbanecm: Continuing with deployment
* 13:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1308106{{!}}fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105{{!}}fix(MentorListCleaner): Do not access property before inicialization (T430689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:58 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet with OS bookworm
* 13:57 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2007.codfw.wmnet with reason: host reimage
* 13:57 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 13:53 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1120.eqiad.wmnet with reason: host reimage
* 13:50 moritzm: installing Linux 5.10.259 on Bullseye hosts
* 13:47 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 13:47 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
* 13:46 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2007.codfw.wmnet with reason: host reimage
* 13:46 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 13:46 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1120.eqiad.wmnet with reason: host reimage
* 13:46 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:46 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:44 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
* 13:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:38 moritzm: installing e2fsprogs updates from Trixie point release
* 13:35 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1308106{{!}}fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105{{!}}fix(MentorListCleaner): Do not access property before inicialization (T430689)]]
* 13:33 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1120.eqiad.wmnet with OS trixie
* 13:33 topranks: reset cr3-eqsin configuration so traffic uses it again after upgrade
* 13:32 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1088.eqiad.wmnet with OS trixie
* 13:32 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS trixie
* 13:32 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns7002.*
* 13:29 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2007.codfw.wmnet with OS trixie
* 13:23 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1004.eqiad.wmnet with reason: host reimage
* 13:20 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2006.codfw.wmnet with OS trixie
* 13:18 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1036.eqiad.wmnet with OS trixie
* 13:17 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1004.eqiad.wmnet with reason: host reimage
* 13:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 13:16 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 13:15 jayme: Istio is being upgraded from 1.24.2 to 1.29.4 on wikikube staging eqiad and codfw - [[phab:T427401|T427401]]
* 13:14 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1087.eqiad.wmnet with OS trixie
* 13:14 topranks: reboot cr3-eqsin to install new JunOS and set PIC 0/0/0 to 100G
* 13:14 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage
* 13:13 jmm@dns1004: END - running authdns-update
* 13:12 jmm@dns1004: START - running authdns-update
* 13:09 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage
* 13:07 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1082.eqiad.wmnet with OS trixie
* 13:07 jmm@dns1004: END - running authdns-update
* 13:06 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1004.eqiad.wmnet with OS bookworm
* 13:05 jmm@dns1004: START - running authdns-update
* 13:02 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2006.codfw.wmnet with reason: host reimage
* 12:58 topranks: load updated JunOS on cr3-eqsin [[phab:T429386|T429386]]
* 12:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage
* 12:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 12:57 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2006.codfw.wmnet with reason: host reimage
* 12:57 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: upgrade JunOS cr3-eqsin
* 12:56 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs[5004-5006].eqsin.wmnet with reason: upgrade JunOS cr3-eqsin
* 12:55 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 12:55 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 12:53 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage
* 12:53 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 12:52 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1088.eqiad.wmnet with OS trixie
* 12:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:52 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:51 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 12:49 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage
* 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 12:48 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage
* 12:44 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1082.eqiad.wmnet with reason: host reimage
* 12:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:42 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 12:42 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 12:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 12:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 12:39 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:39 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: move dumps-nfs IP to the shared one - filippo@cumin1003&quot;
* 12:39 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: move dumps-nfs IP to the shared one - filippo@cumin1003&quot;
* 12:39 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2006.codfw.wmnet with OS trixie
* 12:38 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1082.eqiad.wmnet with reason: host reimage
* 12:36 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 12:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 12:33 filippo@cumin1003: START - Cookbook sre.dns.netbox
* 12:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1036
* 12:32 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1036
* 12:32 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2005.codfw.wmnet with OS trixie
* 12:32 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1087.eqiad.wmnet with OS trixie
* 12:30 jmm@dns1004: END - running authdns-update
* 12:29 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1036
* 12:29 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1036.eqiad.wmnet 21.32.64.10.in-addr.arpa 1.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:29 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1036.eqiad.wmnet 21.32.64.10.in-addr.arpa 1.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:29 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:29 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1036 - blake@cumin1003&quot;
* 12:29 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1036 - blake@cumin1003&quot;
* 12:28 jmm@dns1004: START - running authdns-update
* 12:26 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:26 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:23 blake@cumin1003: START - Cookbook sre.dns.netbox
* 12:23 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1036
* 12:23 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1036.eqiad.wmnet with OS trixie
* 12:22 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet
* 12:22 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1082.eqiad.wmnet with OS trixie
* 12:22 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet
* 12:22 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1036.eqiad.wmnet
* 12:21 marostegui: Restart mariadb@s7 on db1155 to pick up new filters - [[phab:T431124|T431124]]
* 12:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: restarting for replication filter
* 12:20 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:14 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:14 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2005.codfw.wmnet with reason: host reimage
* 12:14 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:08 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:07 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:07 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2005.codfw.wmnet with reason: host reimage
* 12:06 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:06 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:06 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:05 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:05 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:04 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-master-eqiad
* 12:04 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl1002.eqiad.wmnet
* 12:04 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl1002.eqiad.wmnet
* 12:04 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:04 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:03 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:03 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:03 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:03 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 11:59 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl1002.eqiad.wmnet
* 11:59 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl1002.eqiad.wmnet
* 11:59 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:59 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:56 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:56 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:56 klausman@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-master-eqiad
* 11:50 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2005.codfw.wmnet with OS trixie
* 11:49 blake@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on wikikube-worker1160.eqiad.wmnet with reason: Verifying matchers for silence
* 11:42 topranks: cr3-eqsin, begin traffic drain to reset PIC and upgrade JunOS [[phab:T429386|T429386]]
* 11:41 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs[5004-5006].eqsin.wmnet with reason: upgrade JunOS cr3-eqsin
* 11:39 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: upgrade JunOS cr3-eqsin
* 11:36 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=thanos-fe2004.codfw.wmnet
* 11:35 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1086.eqiad.wmnet with OS trixie
* 11:35 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=thanos-fe2004.codfw.wmnet
* 11:32 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1085.eqiad.wmnet with OS trixie
* 11:14 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1086.eqiad.wmnet with reason: host reimage
* 11:10 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1085.eqiad.wmnet with reason: host reimage
* 11:04 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2004.codfw.wmnet with OS trixie
* 11:03 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1086.eqiad.wmnet with reason: host reimage
* 11:02 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1085.eqiad.wmnet with reason: host reimage
* 10:48 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2004.codfw.wmnet with reason: host reimage
* 10:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:46 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1086.eqiad.wmnet with OS trixie
* 10:46 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1085.eqiad.wmnet with OS trixie
* 10:44 cgoubert@deploy1003: Finished deploy [restbase/deploy@2fc37d4]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]] (duration: 16m 44s)
* 10:43 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2004.codfw.wmnet with reason: host reimage
* 10:35 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:27 cgoubert@deploy1003: Started deploy [restbase/deploy@2fc37d4]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]]
* 10:27 cgoubert@deploy1003: Finished deploy [restbase/deploy@8a25036]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]] (duration: 00m 45s)
* 10:26 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1117.eqiad.wmnet with OS trixie
* 10:26 cgoubert@deploy1003: Started deploy [restbase/deploy@8a25036]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]]
* 10:26 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host thanos-fe2004
* 10:25 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host thanos-fe2004
* 10:22 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1116.eqiad.wmnet with OS trixie
* 10:21 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host thanos-fe2004
* 10:21 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) thanos-fe2004.codfw.wmnet 157.32.192.10.in-addr.arpa 7.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:20 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache thanos-fe2004.codfw.wmnet 157.32.192.10.in-addr.arpa 7.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:20 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:20 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host thanos-fe2004 - mvernon@cumin2003&quot;
* 10:20 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host thanos-fe2004 - mvernon@cumin2003&quot;
* 10:15 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2252: Repooling after reboot
* 10:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 10:15 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache
* 10:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2252: Repooling after reboot
* 10:15 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1153.eqiad.wmnet
* 10:14 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1153.eqiad.wmnet
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 10:14 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 10:12 mvernon@cumin2003: START - Cookbook sre.dns.netbox
* 10:12 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host thanos-fe2004
* 10:11 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2004.codfw.wmnet with OS trixie
* 10:07 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1117.eqiad.wmnet with reason: host reimage
* 10:03 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1116.eqiad.wmnet with reason: host reimage
* 09:58 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:58 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1117.eqiad.wmnet with reason: host reimage
* 09:57 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1116.eqiad.wmnet with reason: host reimage
* 09:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 41 days, 15:00:00 on db2252.codfw.wmnet with reason: Security updates
* 09:45 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1117.eqiad.wmnet with OS trixie
* 09:45 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1116.eqiad.wmnet with OS trixie
* 09:28 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1153: Security updates
* 09:28 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 09:28 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 09:28 root@cumin1003: START - Cookbook sre.mysql.depool depool db1153: Security updates
* 09:22 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1016: Security updates
* 09:22 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 09:21 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 09:21 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1016: Security updates
* 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 09:14 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:56 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1016: Security updates
* 08:56 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 08:56 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:56 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1016: Security updates
* 08:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:45 filippo@dns1006: END - running authdns-update
* 08:43 filippo@dns1006: START - running authdns-update
* 08:42 godog: switch dumps-nfs address to be shared with rsync/http - [[phab:T411248|T411248]]
* 08:40 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1016: Security updates
* 08:40 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 08:40 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:40 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1016: Security updates
* 08:29 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host cirrussearch1111.eqiad.wmnet
* 08:29 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:27 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:27 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:25 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:25 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:24 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:17 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:17 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1015: Security updates
* 08:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 08:09 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:09 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1015: Security updates
* 07:42 Msz2001: Deployed private patch for Suggested Ivestigations
* 07:41 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1015: Security updates
* 07:41 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 07:41 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 07:41 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1015: Security updates
* 07:40 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 07:11 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fingerprint warnings - oblivian@cumin1003&quot;
* 07:11 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fingerprint warnings - oblivian@cumin1003
* 07:11 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1024: Security updates
* 07:11 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 07:11 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 07:11 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1024: Security updates
* 07:10 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fingerprint warnings - oblivian@cumin1003
* 07:10 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fingerprint warnings - oblivian@cumin1003&quot;
* 06:55 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host cirrussearch1111.eqiad.wmnet
* 06:55 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 06:48 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1024: Security updates
* 06:48 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 06:48 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 06:48 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1024: Security updates
* 06:42 moritzm: install nginx security updates
* 06:31 root@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool pc1024: Security updates
* 06:21 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1024: Security updates
* 06:19 moritzm: installing php8.2 security updates
* 06:15 moritzm: installing php8.4 security updates
* 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.7 (duration: 02m 38s)
* 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.10  refs [[phab:T430829|T430829]] (duration: 37m 04s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 51s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-06 ==
* 23:30 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306983{{!}}Remove unused user skin preference config (T358273)]], [[gerrit:1306982{{!}}Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921{{!}}Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453{{!}}Drop unused VectorNightMode config (T393977)]] (duration: 09m 39s)
* 23:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1078.eqiad.wmnet with OS trixie
* 23:26 jdlrobson@deploy1003: jdlrobson, bwang: Continuing with deployment
* 23:22 jdlrobson@deploy1003: jdlrobson, bwang: Backport for [[gerrit:1306983{{!}}Remove unused user skin preference config (T358273)]], [[gerrit:1306982{{!}}Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921{{!}}Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453{{!}}Drop unused VectorNightMode config (T393977)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:21 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1306983{{!}}Remove unused user skin preference config (T358273)]], [[gerrit:1306982{{!}}Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921{{!}}Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453{{!}}Drop unused VectorNightMode config (T393977)]]
* 23:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1078.eqiad.wmnet with reason: host reimage
* 23:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1078.eqiad.wmnet with reason: host reimage
* 22:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1078.eqiad.wmnet with OS trixie
* 22:29 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cirrussearch1114.eqiad.wmnet with reason: reimage on hold until restore completes
* 22:22 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cirrussearch[1079,1115].eqiad.wmnet with reason: reimage on hold until restore completes
* 21:18 maryum: Deployed security fix for [[phab:T428006|T428006]]
* 20:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1079.eqiad.wmnet with OS trixie
* 20:33 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1077.eqiad.wmnet with OS trixie
* 20:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1115.eqiad.wmnet with OS trixie
* 20:25 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1079.eqiad.wmnet with reason: host reimage
* 20:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1079.eqiad.wmnet with reason: host reimage
* 20:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1303490{{!}}T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812{{!}}Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] (duration: 08m 14s)
* 20:14 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1077.eqiad.wmnet with reason: host reimage
* 20:10 krinkle@deploy1003: krinkle, pushpaktiwari: Continuing with deployment
* 20:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1115.eqiad.wmnet with reason: host reimage
* 20:08 krinkle@deploy1003: krinkle, pushpaktiwari: Backport for [[gerrit:1303490{{!}}T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812{{!}}Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1077.eqiad.wmnet with reason: host reimage
* 20:06 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1303490{{!}}T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812{{!}}Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]]
* 20:05 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1079.eqiad.wmnet with OS trixie
* 20:04 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1115.eqiad.wmnet with reason: host reimage
* 19:52 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1077.eqiad.wmnet with OS trixie
* 19:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1115.eqiad.wmnet with OS trixie
* 19:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 19:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 18:59 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1114.eqiad.wmnet with OS trixie
* 18:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1114.eqiad.wmnet with reason: host reimage
* 18:35 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1114.eqiad.wmnet with reason: host reimage
* 18:32 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1112.eqiad.wmnet with OS trixie
* 18:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1114.eqiad.wmnet with OS trixie
* 18:21 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1072.eqiad.wmnet with OS trixie
* 18:14 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1112.eqiad.wmnet with reason: host reimage
* 18:10 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1112.eqiad.wmnet with reason: host reimage
* 17:59 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1072.eqiad.wmnet with reason: host reimage
* 17:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1112.eqiad.wmnet with OS trixie
* 17:55 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1072.eqiad.wmnet with reason: host reimage
* 17:39 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1072.eqiad.wmnet with OS trixie
* 17:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1071.eqiad.wmnet with OS trixie
* 17:18 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1070.eqiad.wmnet with OS trixie
* 17:16 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1084.eqiad.wmnet with OS trixie
* 16:54 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1071.eqiad.wmnet with reason: host reimage
* 16:53 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1084.eqiad.wmnet with reason: host reimage
* 16:51 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1070.eqiad.wmnet with reason: host reimage
* 16:49 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1084.eqiad.wmnet with reason: host reimage
* 16:38 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1071.eqiad.wmnet with OS trixie
* 16:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1096.eqiad.wmnet with OS trixie
* 16:35 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1070.eqiad.wmnet with OS trixie
* 16:33 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1084.eqiad.wmnet with OS trixie
* 16:30 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1089.eqiad.wmnet with OS trixie
* 16:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1103.eqiad.wmnet with OS trixie
* 16:19 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1096.eqiad.wmnet with reason: host reimage
* 16:14 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1096.eqiad.wmnet with reason: host reimage
* 16:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1089.eqiad.wmnet with reason: host reimage
* 16:05 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage
* 16:02 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1003.eqiad.wmnet with OS bookworm
* 16:01 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1089.eqiad.wmnet with reason: host reimage
* 16:00 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage
* 15:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1096.eqiad.wmnet with OS trixie
* 15:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1080.eqiad.wmnet with OS trixie
* 15:46 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1089.eqiad.wmnet with OS trixie
* 15:45 dancy@deploy1003: Installation of scap version &quot;4.272.0&quot; completed for 158 hosts
* 15:43 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1103.eqiad.wmnet with OS trixie
* 15:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1113.eqiad.wmnet with OS trixie
* 15:41 dancy@deploy1003: Installing scap version &quot;4.272.0&quot; for 158 host(s)
* 15:40 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 15:39 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 15:38 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 15:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1069.eqiad.wmnet with OS trixie
* 15:37 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 15:36 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:34 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 15:33 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage
* 15:27 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage
* 15:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1113.eqiad.wmnet with reason: host reimage
* 15:19 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1069.eqiad.wmnet with reason: host reimage
* 15:18 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1113.eqiad.wmnet with reason: host reimage
* 15:16 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1069.eqiad.wmnet with reason: host reimage
* 15:11 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:11 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1080.eqiad.wmnet with OS trixie
* 15:11 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:05 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1113.eqiad.wmnet with OS trixie
* 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1003.eqiad.wmnet with reason: host reimage
* 14:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1003.eqiad.wmnet with OS bookworm
* 14:33 elukey: rolled out spicerack on all cumin nodes - [[phab:T429699|T429699]]
* 14:32 elukey: upgrade all bookworm hosts to pywmflib 3.1 - [[phab:T430552|T430552]]
* 14:14 marostegui: Setup x4 eqiad topology [[phab:T404715|T404715]]
* 14:13 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 14:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db2230.codfw.wmnet
* 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db2230.codfw.wmnet
* 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[2001-2002].codfw.wmnet
* 13:51 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 13:45 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97)
* 13:45 cwilliams@cumin1003: dbmaint on s4@codfw [[phab:T429893|T429893]]
* 13:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 13:42 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-master-codfw
* 13:42 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl2002.codfw.wmnet
* 13:42 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl2002.codfw.wmnet
* 13:38 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl2002.codfw.wmnet
* 13:38 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl2002.codfw.wmnet
* 13:38 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl2001.codfw.wmnet
* 13:38 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl2001.codfw.wmnet
* 13:35 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl2001.codfw.wmnet
* 13:35 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl2001.codfw.wmnet
* 13:35 klausman@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-master-codfw
* 12:30 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1265672{{!}}robots.php: Change Beta Cluster override from prepend to replace]] (duration: 25m 11s)
* 12:24 krinkle@deploy1003: krinkle: Continuing with deployment
* 12:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet
* 12:09 krinkle@deploy1003: krinkle: Backport for [[gerrit:1265672{{!}}robots.php: Change Beta Cluster override from prepend to replace]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:08 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2048.codfw.wmnet
* 12:05 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1265672{{!}}robots.php: Change Beta Cluster override from prepend to replace]]
* 11:57 moritzm: installing curl security updates
* 11:49 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 11:31 moritzm: installing nano security updates
* 11:07 moritzm: failover Ganeti master in codfw to ganeti2032 [[phab:T430909|T430909]]
* 11:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 11:04 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1005.eqiad.wmnet with OS trixie
* 11:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:50 jmm@dns1004: END - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:46 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:44 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1005.eqiad.wmnet with reason: host reimage
* 10:38 elukey@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1005.eqiad.wmnet with reason: host reimage
* 10:31 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:31 marostegui: Setup x4 codfw topology [[phab:T404715|T404715]]
* 10:31 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:24 elukey: spicerack 13.0.0 deployed on cumin2002
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:23 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:23 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:21 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1005.eqiad.wmnet with OS trixie
* 10:20 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:17 elukey: uploaded spicerack_13.0.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia
* 09:54 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:52 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:20 elukey: upgrade all bullseye hosts to pywmflib 3.1 - [[phab:T430552|T430552]]
* 09:10 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s4
* 09:10 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s6
* 09:07 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 08:58 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 08:56 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 08:56 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 08:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2002.codfw.wmnet
* 08:06 godog: remove cloudvirt1046, cloudvirt1062, cloudvirt1074, cloudvirt1075 from maintenance aggregate and put them in network-ovs - [[phab:T424802|T424802]]
* 08:00 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet
* 07:58 hashar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307617{{!}}Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] (duration: 32m 53s)
* 07:57 fabfur: repooled cp4038
* 07:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp4038.*
* 07:53 moritzm: installing pyjwt security updates
* 07:47 moritzm: installing openjpeg2 security updates
* 07:45 hashar@deploy1003: vadymts1, hashar: Continuing with deployment
* 07:43 hashar@deploy1003: vadymts1, hashar: Backport for [[gerrit:1307617{{!}}Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:38 moritzm: installing python-urllib3 security updates
* 07:37 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
* 07:30 fabfur: depooled cp4038 to investigate on possible maxmind failure
* 07:30 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp4038.*
* 07:30 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp4038.*
* 07:29 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw
* 07:25 hashar@deploy1003: Started scap sync-world: Backport for [[gerrit:1307617{{!}}Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]]
* 06:13 moritzm: installing Linux 6.12.95 on trixie hosts
* 05:20 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s6
* 05:20 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s4
* 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1015.eqiad.wmnet with reason: cloning
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 08s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-05 ==
* 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 08s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-04 ==
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 58s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-03 ==
* 17:08 topranks: revert protocol preference changes on cr3-ulsfo after upgrade
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr2-eqord with reason: upgrade JunOS cr3-ulsfo
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr4-ulsfo with reason: upgrade JunOS cr3-ulsfo
* 16:48 topranks: reboot cr3-ulsfo to upgrade JunOS and reset linecard [[phab:T424839|T424839]]
* 15:52 topranks: adjust outbound BGP policies on cr3-ulsfo to drain router of traffic [[phab:T424839|T424839]]
* 15:45 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs[4008-4010].ulsfo.wmnet with reason: upgrade JunOS cr3-ulsfo
* 15:44 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw1-[22-23]-ulsfo,cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: upgrade JunOS cr3-ulsfo
* 15:36 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 14:40 cmooney@dns3003: END - running authdns-update
* 14:26 cmooney@dns3003: START - running authdns-update
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:19 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:16 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 13:38 sukhe@dns1004: END - running authdns-update
* 13:35 sukhe@dns1004: START - running authdns-update
* 13:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:26 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:18 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:17 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:14 moritzm: imported samplicator 1.3.8rc1-1+deb13u1 to trixie-wikimedia/main [[phab:T337208|T337208]]
* 13:13 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:07 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:07 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:02 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:02 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:00 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:58 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:53 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:52 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:50 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 12:50 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 12:47 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:41 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:39 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:32 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 12:26 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 12:23 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 12:19 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[2004-2007].codfw.wmnet
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:15 jynus@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:09 jynus@cumin2003: START - Cookbook sre.dns.netbox
* 11:58 jynus@cumin2003: START - Cookbook sre.hosts.decommission for hosts backup[2004-2007].codfw.wmnet
* 10:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[1004-1007].eqiad.wmnet
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 10:01 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 09:52 jynus@cumin1003: START - Cookbook sre.dns.netbox
* 09:39 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:36 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts backup[1004-1007].eqiad.wmnet
* 09:36 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 09:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 09:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:57 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:55 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 08:49 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 08:49 atsukoito: depooling cirrussearch in codfw because of regression after upgrade [[phab:T431091|T431091]]
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mirror1001.wikimedia.org
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:29 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:18 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 08:11 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts mirror1001.wikimedia.org
* 06:15 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 18s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-02 ==
* 22:55 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1003.wikimedia.org with OS trixie
* 22:29 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:23 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:05 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1003.wikimedia.org with OS trixie
* 22:03 mutante: contint1003 (zuul.wikimedia.org) - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 22:03 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on zuul.wikimedia.org with reason: reimage
* 21:39 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 18s)
* 21:39 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 21:20 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 21:19 sbassett: Deployed security fix for [[phab:T428829|T428829]]
* 20:58 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:55 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:40 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] (duration: 12m 35s)
* 20:36 arlolra@deploy1003: cscott, arlolra: Continuing with deployment
* 20:35 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 20s)
* 20:35 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 20:33 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint2003.wikimedia.org with OS trixie
* 20:31 arlolra@deploy1003: cscott, arlolra: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Cha
* 20:28 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]]
* 20:17 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] (duration: 08m 13s)
* 20:14 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:13 sbassett@deploy1003: sbassett: Continuing with deployment
* 20:11 sbassett@deploy1003: sbassett: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:09 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]]
* 20:08 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 20:08 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 20:08 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:05 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 19:49 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint2003.wikimedia.org with OS trixie
* 19:48 mutante: contint2003 - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 18:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:13 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 17:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1003.eqiad.wmnet with OS bookworm
* 17:52 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:52 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:51 jasmine@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=wikikube-ctrl1005.eqiad.wmnet
* 17:48 jasmine_: homer &quot;cr*eqiad*&quot; commit &quot;Added new stacked control plane wikikube-ctrl1005&quot;
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
* 17:31 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] (duration: 09m 33s)
* 17:26 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 17:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:23 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:21 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]]
* 17:18 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:16 rscout@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:15 rscout@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:12 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wcqs[2002-2003].codfw.wmnet,wcqs1002.eqiad.wmnet with reason: reimaging hosts
* 17:08 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 17:07 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 17:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:03 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:03 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1003
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1003
* 17:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1003.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:57 tappof: bump space for prometheus k8s-aux in eqiad
* 16:55 cmooney@dns3003: END - running authdns-update
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:53 cmooney@dns3003: START - running authdns-update
* 16:52 ryankemper: [ml-serve-eqiad] Cleared out 1302 failed (Evicted) pods: `kubectl -n llm delete pods --field-selector=status.phase=Failed`, freeing calico-kube-controllers from OOM crashloop (evictions were caused by disk pressure)
* 16:49 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:46 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 16:39 rzl@dns1004: END - running authdns-update
* 16:37 rzl@dns1004: START - running authdns-update
* 16:36 rzl@dns1004: START - running authdns-update
* 16:35 rzl@deploy1003: Finished scap sync-world: [[phab:T416623|T416623]] (duration: 10m 19s)
* 16:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:33 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:30 rzl@deploy1003: rzl: Continuing with deployment
* 16:28 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:26 rzl@deploy1003: rzl: [[phab:T416623|T416623]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:25 rzl@deploy1003: Started scap sync-world: [[phab:T416623|T416623]]
* 16:25 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: sync
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: sync
* 16:16 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1003.eqiad.wmnet with OS bookworm
* 16:11 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:11 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 16:08 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 16:08 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Security updates
* 15:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:45 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:45 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Security updates
* 15:42 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1003.eqiad.wmnet with OS bookworm
* 15:24 moritzm: installing busybox updates from bookworm point release
* 15:20 moritzm: installing busybox updates from trixie point release
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:15 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:15 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Security updates
* 15:13 moritzm: installing giflib security updates
* 15:08 moritzm: installing Tomcat security updates
* 14:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 14:56 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 14:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:53 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 14:53 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 14:53 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Security updates
* 14:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:52 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94711 and previous config saved to /var/cache/conftool/dbconfig/20260702-144644-fceratto.json
* 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94709 and previous config saved to /var/cache/conftool/dbconfig/20260702-143636-fceratto.json
* 14:32 moritzm: installing libdbi-perl security updates
* 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94708 and previous config saved to /var/cache/conftool/dbconfig/20260702-142628-fceratto.json
* 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94707 and previous config saved to /var/cache/conftool/dbconfig/20260702-141621-fceratto.json
* 14:12 moritzm: installing rsync security updates
* 14:11 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94706 and previous config saved to /var/cache/conftool/dbconfig/20260702-140959-fceratto.json
* 14:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2205.codfw.wmnet with reason: Maintenance
* 14:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 14:07 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 Tran: Deployed patch for [[phab:T427287|T427287]]
* 14:04 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:59 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 13:59 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2205 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94704 and previous config saved to /var/cache/conftool/dbconfig/20260702-135505-fceratto.json
* 13:54 moritzm: installing sed security updates
* 13:53 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:52 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2209 to s3 primary [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94703 and previous config saved to /var/cache/conftool/dbconfig/20260702-135235-fceratto.json
* 13:52 federico3: Starting s3 codfw failover from db2205 to db2209 - [[phab:T430912|T430912]]
* 13:51 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:51 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2209 with weight 0 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94702 and previous config saved to /var/cache/conftool/dbconfig/20260702-134719-fceratto.json
* 13:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [[phab:T430912|T430912]]
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:40 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:37 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 13:36 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:36 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:34 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 13:30 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:26 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:25 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
* 13:23 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:22 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 13:17 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:17 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org
* 13:12 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart (exit_code=97) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=97) rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:09 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] (duration: 07m 20s)
* 13:05 aude@deploy1003: jdrewniak, aude: Continuing with deployment
* 13:04 aude@deploy1003: jdrewniak, aude: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:02 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]]
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs-categories1001.eqiad.wmnet
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 12:10 jmm@dns1004: END - running authdns-update
* 12:07 jmm@dns1004: START - running authdns-update
* 11:51 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 11:44 btullis@cumin1003: START - Cookbook sre.dns.netbox
* 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:42 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 11:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 11:39 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts wdqs-categories1001.eqiad.wmnet
* 11:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 11:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 11:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 11:29 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:29 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 10:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Repooling
* 10:49 jmm@dns1004: END - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94698 and previous config saved to /var/cache/conftool/dbconfig/20260702-103146-fceratto.json
* 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94696 and previous config saved to /var/cache/conftool/dbconfig/20260702-102137-fceratto.json
* 10:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1017.eqiad.wmnet
* 10:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 10:18 fceratto@cumin1003: Removing es1033 from zarcillo [[phab:T408772|T408772]]
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1033.eqiad.wmnet
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:14 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:13 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1017.eqiad.wmnet
* 10:12 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling
* 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94693 and previous config saved to /var/cache/conftool/dbconfig/20260702-101130-fceratto.json
* 10:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:10 fceratto@cumin1003: START - Cookbook sre.dns.netbox
* 10:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:03 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts es1033.eqiad.wmnet
* 10:03 fceratto@cumin1003: START - Cookbook sre.mysql.decommission
* 10:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94691 and previous config saved to /var/cache/conftool/dbconfig/20260702-100122-fceratto.json
* 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94690 and previous config saved to /var/cache/conftool/dbconfig/20260702-095529-fceratto.json
* 09:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 09:54 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:53 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover
* 09:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:44 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2213 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94688 and previous config saved to /var/cache/conftool/dbconfig/20260702-093859-fceratto.json
* 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94687 and previous config saved to /var/cache/conftool/dbconfig/20260702-093650-fceratto.json
* 09:36 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T430923|T430923]]
* 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94686 and previous config saved to /var/cache/conftool/dbconfig/20260702-093004-fceratto.json
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2192 with weight 0 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94685 and previous config saved to /var/cache/conftool/dbconfig/20260702-092455-fceratto.json
* 09:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 [[phab:T430923|T430923]]
* 09:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94684 and previous config saved to /var/cache/conftool/dbconfig/20260702-091957-fceratto.json
* 09:16 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] (duration: 06m 57s)
* 09:13 moritzm: installing libgcrypt20 security updates
* 09:12 kharlan@deploy1003: kharlan: Continuing with deployment
* 09:11 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94683 and previous config saved to /var/cache/conftool/dbconfig/20260702-090950-fceratto.json
* 09:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]]
* 09:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:01 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] (duration: 07m 07s)
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94682 and previous config saved to /var/cache/conftool/dbconfig/20260702-085942-fceratto.json
* 08:57 kharlan@deploy1003: kharlan: Continuing with deployment
* 08:56 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]]
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94681 and previous config saved to /var/cache/conftool/dbconfig/20260702-085237-fceratto.json
* 08:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 08:43 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:40 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:25 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] (duration: 11m 44s)
* 08:21 cscott@deploy1003: cscott: Continuing with deployment
* 08:16 cscott@deploy1003: cscott: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:14 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]]
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Migration of db1244.eqiad.wmnet completed
* 08:02 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:02 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:01 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] (duration: 18m 58s)
* 08:01 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:01 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 07:59 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:59 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2006.wikimedia.org
* 07:58 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:57 cscott@deploy1003: cscott: Continuing with deployment
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:54 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2006.wikimedia.org
* 07:54 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:54 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:44 cscott@deploy1003: cscott: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 07:44 moritzm: installing node-lodash security updates
* 07:42 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]]
* 07:39 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 07:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] (duration: 07m 28s)
* 07:26 cscott@deploy1003: ssastry, cscott: Continuing with deployment
* 07:25 cscott@deploy1003: ssastry, cscott: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:23 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1244: Migration of db1244.eqiad.wmnet completed
* 07:22 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]]
* 07:16 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] (duration: 06m 55s)
* 07:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1244.eqiad.wmnet with OS trixie
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:09 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]]
* 06:54 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:50 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1250.eqiad.wmnet with OS trixie
* 06:34 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1244.eqiad.wmnet with OS trixie
* 06:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: dbmaint on s4@eqiad [[phab:T429893|T429893]]
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:14 cwilliams@dns1006: END - running authdns-update
* 06:12 cwilliams@dns1006: START - running authdns-update
* 06:11 cwilliams@dns1006: END - running authdns-update
* 06:11 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1244 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94676 and previous config saved to /var/cache/conftool/dbconfig/20260702-061059-cwilliams.json
* 06:09 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:09 cwilliams@dns1006: START - running authdns-update
* 06:08 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0)  on VRTS host vrts1003.eqiad.wmnet
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1160 to s4 primary and set section read-write [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94675 and previous config saved to /var/cache/conftool/dbconfig/20260702-060746-cwilliams.json
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94674 and previous config saved to /var/cache/conftool/dbconfig/20260702-060704-cwilliams.json
* 06:06 cezmunsta: Starting s4 eqiad failover from db1244 to db1160 - [[phab:T430817|T430817]]
* 06:04 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade  on VRTS host vrts1003.eqiad.wmnet
* 05:59 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1160 with weight 0 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94673 and previous config saved to /var/cache/conftool/dbconfig/20260702-055927-cwilliams.json
* 05:59 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430817|T430817]]
* 05:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1250.eqiad.wmnet with OS trixie
* 05:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1250.eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 05:39 marostegui: Failover m3 (phabricator) from db1250 to db1228 - [[phab:T430158|T430158]]
* 05:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 04:45 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] (duration: 09m 08s)
* 04:41 tstarling@deploy1003: tstarling, reedy: Continuing with deployment
* 04:38 tstarling@deploy1003: tstarling, reedy: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:36 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]]
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 59s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:16 ryankemper: [[phab:T429844|T429844]] [opensearch] completed `cirrussearch2111` reimage; all codfw search clusters are green, all nodes now report `OpenSearch 2.19.5`, and the temporary chi voting exclusion has been removed
* 00:57 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:29 ryankemper: [[phab:T429844|T429844]] [opensearch] depooled codfw search-omega/search-psi discovery records to match existing codfw search depool during OpenSearch 2.19 migration
* 00:29 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 00:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:01 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:00 ryankemper: [[phab:T429844|T429844]] [opensearch] chi cluster recovered after stopping `opensearch_1@production-search-codfw` on `cirrussearch2111`

== 2026-07-01 ==
* 23:59 ryankemper: [[phab:T429844|T429844]] [opensearch] stopped `opensearch_1@production-search-codfw` on `cirrussearch2111` after chi cluster-manager election churn following `voting_config_exclusions` POST; hoping this triggers a re-election
* 23:52 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 23:50 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2003.codfw.wmnet with OS bookworm
* 22:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 22:13 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2084.codfw.wmnet with OS trixie
* 22:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 22:03 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:01 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 21:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 21:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:36 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:35 bking@cumin2003: START - Cookbook sre.dns.netbox
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2003
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2003.codfw.wmnet with OS bookworm
* 21:19 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2084.codfw.wmnet with OS trixie
* 21:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2108.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:45 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:28 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 20:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 19:59 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2108.codfw.wmnet with OS trixie
* 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2093.codfw.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:43 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2080.codfw.wmnet with OS trixie
* 19:28 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:18 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:17 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 19:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 18:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2093.codfw.wmnet with OS trixie
* 18:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2080.codfw.wmnet with OS trixie
* 18:27 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 18:18 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] (duration: 09m 15s)
* 18:13 jgiannelos@deploy1003: jgiannelos, neriah: Continuing with deployment
* 18:11 jgiannelos@deploy1003: jgiannelos, neriah: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:09 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]]
* 17:40 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 16:58 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 30 hosts
* 16:57 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 30 hosts
* 16:52 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2202.codfw.wmnet
* 16:52 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2202.codfw.wmnet
* 16:51 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2012.codfw.wmnet
* 16:51 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2012.codfw.wmnet
* 16:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2076.codfw.wmnet with OS trixie
* 16:49 brett: Start pybal on lvs2012 - [[phab:T429861|T429861]]
* 16:49 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.remove-downtime (exit_code=97) for 59 hosts
* 16:48 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 59 hosts
* 16:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2061.codfw.wmnet with OS trixie
* 16:30 dancy@deploy1003: Installation of scap version &quot;4.271.0&quot; completed for 2 hosts
* 16:28 dancy@deploy1003: Installing scap version &quot;4.271.0&quot; for 2 host(s)
* 16:23 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:18 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:18 jasmine@dns1004: END - running authdns-update
* 16:16 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:15 jasmine@dns1004: START - running authdns-update
* 16:14 jasmine@dns1004: END - running authdns-update
* 16:12 jasmine@dns1004: START - running authdns-update
* 16:07 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: maintenance
* 16:06 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt with reason: Junos upograde
* 16:00 papaul: ongoing maintenance on lsw1-b2-codfw
* 16:00 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2076.codfw.wmnet with OS trixie
* 15:59 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:59 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:57 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2061.codfw.wmnet with OS trixie
* 15:55 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:55 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:51 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:51 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Repooling after switchover
* 15:50 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:50 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 15:48 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2092.codfw.wmnet with OS trixie
* 15:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:30 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 15:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:22 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: Rack B2 maintenance - [[phab:T429861|T429861]]
* 15:21 brett: Stopping pybal on lvs2012 in preparation for codfw rack b2 maintenance - [[phab:T429861|T429861]]
* 15:20 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:12 _joe_: restarted manually alertmanager-irc-relay
* 15:12 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt with reason: Junos upograde
* 15:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 15:07 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:06 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:02 papaul: ongoing maintenance on lsw1-a8-codfw
* 14:31 topranks: POWERING DOWN CR1-EQIAD for line card installation [[phab:T426343|T426343]]
* 14:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] (duration: 08m 57s)
* 14:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
* 14:24 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:22 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]]
* 14:22 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:16 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:15 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:14 topranks: re-enable routing-engine graceful-failover on cr1-eqiad [[phab:T417873|T417873]]
* 14:13 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:13 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2220: Repooling after switchover
* 14:12 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:11 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:08 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] (duration: 10m 01s)
* 14:07 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2220 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94664 and previous config saved to /var/cache/conftool/dbconfig/20260701-140729-fceratto.json
* 14:06 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2159 to s7 primary [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94663 and previous config saved to /var/cache/conftool/dbconfig/20260701-140503-fceratto.json
* 14:04 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:04 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 14:04 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 14:04 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:04 dreamyjazz@deploy1003: anzx, dreamyjazz: Continuing with deployment
* 14:04 federico3: Starting s7 codfw failover from db2220 to db2159 - [[phab:T430826|T430826]]
* 14:03 jmm@dns1004: END - running authdns-update
* 14:03 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:03 topranks: flipping cr1-eqiad active routing-enginer back to RE0 [[phab:T417873|T417873]]
* 14:03 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudsw1-c8-eqiad,cloudsw1-d5-eqiad with reason: router upgrades eqiad
* 14:01 jmm@dns1004: START - running authdns-update
* 14:00 dreamyjazz@deploy1003: anzx, dreamyjazz: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2159 with weight 0 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94662 and previous config saved to /var/cache/conftool/dbconfig/20260701-135906-fceratto.json
* 13:58 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]]
* 13:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 [[phab:T430826|T430826]]
* 13:56 topranks: reboot routing-enginer RE0 on cr1-eqiad [[phab:T417873|T417873]]
* 13:48 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1006.wikimedia.org
* 13:44 atsuko@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:43 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1006.wikimedia.org
* 13:41 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1005.wikimedia.org
* 13:37 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1005.wikimedia.org
* 13:37 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on pfw1-eqiad with reason: router upgrades eqiad
* 13:35 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[1017-1020].eqiad.wmnet with reason: router upgrades eqiad
* 13:34 caro@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] (duration: 07m 59s)
* 13:30 caro@deploy1003: caro: Continuing with deployment
* 13:28 caro@deploy1003: caro: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:27 topranks: route-engine failover cr1-eqiad
* 13:26 caro@deploy1003: Started scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]]
* 13:15 topranks: rebooting routing-engine 1 on cr1-eqiad [[phab:T417873|T417873]]
* 13:13 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] (duration: 08m 29s)
* 13:13 moritzm: installing qemu security updates
* 13:11 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 13:11 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 13:09 jgiannelos@deploy1003: jgiannelos: Continuing with deployment
* 13:08 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:07 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:06 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance
* 13:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2214: Repooling after switchover
* 13:05 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]]
* 13:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling after switchover
* 13:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2214 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94660 and previous config saved to /var/cache/conftool/dbconfig/20260701-130413-fceratto.json
* 13:01 moritzm: installing python3.13 security updates
* 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2229 to s6 primary [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94659 and previous config saved to /var/cache/conftool/dbconfig/20260701-125959-fceratto.json
* 12:59 federico3: Starting s6 codfw failover from db2214 to db2229 - [[phab:T430814|T430814]]
* 12:57 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 13 hosts with reason: router upgrade and line card install
* 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2229 with weight 0 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94658 and previous config saved to /var/cache/conftool/dbconfig/20260701-125149-fceratto.json
* 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Primary switchover s6 [[phab:T430814|T430814]]
* 12:50 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet
* 12:50 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet
* 12:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2100.codfw.wmnet with OS trixie
* 12:38 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2083.codfw.wmnet with OS trixie
* 12:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Migration of db2240.codfw.wmnet completed
* 12:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:00 topranks: drain traffic on cr1-eqiad to allow for line card install and JunOS upgrade [[phab:T426343|T426343]]
* 11:52 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2083.codfw.wmnet with OS trixie
* 11:50 cmooney@dns2005: END - running authdns-update
* 11:49 cmooney@dns2005: START - running authdns-update
* 11:48 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2100.codfw.wmnet with OS trixie
* 11:40 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 11:40 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 11:31 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2240: Migration of db2240.codfw.wmnet completed
* 11:30 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:28 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
* 11:27 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:23 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2240.codfw.wmnet with OS trixie
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:17 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:16 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:16 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:15 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2086.codfw.wmnet with OS trixie
* 11:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2106.codfw.wmnet with OS trixie
* 11:14 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:13 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:12 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:09 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2115.codfw.wmnet with OS trixie
* 11:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 11:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 10:53 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:44 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:44 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2240.codfw.wmnet with OS trixie
* 10:44 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:42 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: dbmaint on s4@codfw [[phab:T429893|T429893]]
* 10:40 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 10:39 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:27 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2240 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94653 and previous config saved to /var/cache/conftool/dbconfig/20260701-102658-cwilliams.json
* 10:26 moritzm: installing nginx security updates
* 10:26 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2086.codfw.wmnet with OS trixie
* 10:23 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2179 to s4 primary [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94652 and previous config saved to /var/cache/conftool/dbconfig/20260701-102356-cwilliams.json
* 10:23 cezmunsta: Starting s4 codfw failover from db2240 to db2179 - [[phab:T430127|T430127]]
* 10:23 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2106.codfw.wmnet with OS trixie
* 10:20 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2115.codfw.wmnet with OS trixie
* 10:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2179 with weight 0 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94651 and previous config saved to /var/cache/conftool/dbconfig/20260701-101531-cwilliams.json
* 10:15 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430127|T430127]]
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 09:39 mszwarc@deploy1003: Synchronized private/SuggestedInvestigationsSignals/SuggestedInvestigationsSignal4n.php: Update SI signal 4n (duration: 06m 08s)
* 09:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:21 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:54 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:38 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:38 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:36 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:21 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] (duration: 36m 11s)
* 08:15 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:09 mszwarc@deploy1003: mszwarc, abi: Continuing with deployment
* 08:03 mszwarc@deploy1003: mszwarc, abi: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:55 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 07:51 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 07:45 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]]
* 07:30 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050] (duration: 00m 22s)
* 07:29 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050]
* 07:28 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050] (duration: 01m 59s)
* 07:28 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] (duration: 07m 19s)
* 07:26 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050]
* 07:26 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050] (duration: 04m 32s)
* 07:24 mszwarc@deploy1003: wmde-fisch, mszwarc: Continuing with deployment
* 07:23 mszwarc@deploy1003: wmde-fisch, mszwarc: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:21 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050]
* 07:21 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050] (duration: 02m 01s)
* 07:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]]
* 07:19 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050]
* 07:13 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] (duration: 09m 13s)
* 07:09 mszwarc@deploy1003: mszwarc, chlod, revi: Continuing with deployment
* 07:06 mszwarc@deploy1003: mszwarc, chlod, revi: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:04 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]]
* 06:55 elukey: upgrade all trixie hosts to pywmflib 3.0 - [[phab:T430552|T430552]]
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:31 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Niharika29 out of all services on: 2453 hosts
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.hiddenparma (exit_code=99) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:01 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1039.eqiad.wmnet with reason: issues
* 05:41 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1027.eqiad.wmnet
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2068.codfw.wmnet with OS trixie
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s2
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s7
* 05:36 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:16 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:15 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:09 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2067.codfw.wmnet with OS trixie
* 04:56 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2068.codfw.wmnet with OS trixie
* 04:49 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:45 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:27 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2067.codfw.wmnet with OS trixie
* 03:47 slyngshede@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Hardware crash
* 03:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:59 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:55 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:35 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:31 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:30 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:26 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:09 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 54s)
* 02:03 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:07 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad back to read-write - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94649 and previous config saved to /var/cache/conftool/dbconfig/20260701-010716-ladsgroup.json
* 01:05 ladsgroup@dns1004: END - running authdns-update
* 01:05 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool es1039 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94648 and previous config saved to /var/cache/conftool/dbconfig/20260701-010551-ladsgroup.json
* 01:03 ladsgroup@dns1004: START - running authdns-update
* 01:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Promote es1035 to es7 primary [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94647 and previous config saved to /var/cache/conftool/dbconfig/20260701-010002-ladsgroup.json
* 00:58 Amir1: Starting es7 eqiad failover from es1039 to es1035 - [[phab:T430765|T430765]]
* 00:53 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es1035 with weight 0 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94646 and previous config saved to /var/cache/conftool/dbconfig/20260701-005329-ladsgroup.json
* 00:53 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Primary switchover es7 [[phab:T430765|T430765]]
* 00:42 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad as read-only for maintenance - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94645 and previous config saved to /var/cache/conftool/dbconfig/20260701-004221-ladsgroup.json
* 00:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2102.codfw.wmnet with OS trixie
* 00:15 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2103.codfw.wmnet with OS trixie
* 00:05 dr0ptp4kt: DEPLOYED Refinery at {{Gerrit|4e7a2b32}} for changes: pageview allowlist {{Gerrit|1305158}} (+min.wikiquote) {{Gerrit|1305162}} (+bol.wikipedia), {{Gerrit|1305156}} (+isv.wikipedia); {{Gerrit|1305980}} (pv allowlist -api.wikimedia, sqoop +isvwiki); sqoop {{Gerrit|1295064}} (+globalimagelinks) {{Gerrit|1295069}} (+filerevision) using scap, then deployed onto HDFS (manual copyToLocal required additionally)

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>g73lyduu6muibg4iyr02spptgh67h3s</sha1>
    </revision>
    <revision>
      <id>2435523</id>
      <parentid>2435522</parentid>
      <timestamp>2026-07-12T02:01:29Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 01m 17s)</comment>
      <origin>2435523</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="329933" sha1="1al9ufrjcm816zmqqcgb4ceozwu7adw" xml:space="preserve">
== 2026-07-12 ==
* 02:01 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 01m 17s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-11 ==
* 02:06 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 26s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-10 ==
* 19:12 jhathaway@dns1004: END - running authdns-update
* 19:10 jhathaway@dns1004: START - running authdns-update
* 18:23 mutante: vrts2002 rebooting (not the active host)
* 18:21 mutante: lists2001, phab2003 - rebooting (not the active hosts)
* 18:16 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic2-codfw
* 18:15 swfrench@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic2-codfw
* 17:15 mutante: [doc1004:~] $ sudo systemctl start rsync-doc-host-data-sync ([[phab:T431856|T431856]])
* 17:09 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1004.eqiad.wmnet
* 17:08 jhathaway@dns1004: END - running authdns-update
* 17:07 jhathaway@dns1004: START - running authdns-update
* 17:06 jhathaway: depooling puppetserver1002, cause of errors is still unknown
* 17:03 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1004.eqiad.wmnet
* 16:57 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1003.eqiad.wmnet
* 16:51 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1003.eqiad.wmnet
* 16:48 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 16:48 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2004.codfw.wmnet
* 16:42 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2004.codfw.wmnet
* 16:41 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2003.codfw.wmnet
* 16:35 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2003.codfw.wmnet
* 16:33 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2002.codfw.wmnet
* 16:27 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2002.codfw.wmnet
* 16:25 mutante: gitlab-runners (production) rebooting cluster one by one
* 16:17 mutante: etherpad1004/etherpad2002 - (etherpad.wikimedia.org) - rebooting
* 16:13 mutante: doc1004/doc2003 (doc.wikimedia.org backends) - rebooting
* 16:02 mutante: releases1003/releases2003 (releases.wikimedia.org backends) - rebooting for maintenance
* 15:26 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2007-dev.codfw.wmnet
* 15:19 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet
* 15:14 filippo@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcephosd2007-dev.codfw.wmnet
* 15:14 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet
* 15:14 filippo@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudcephosd2006-dev.codfw.wmnet
* 15:07 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2006-dev.codfw.wmnet
* 15:07 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2005-dev.codfw.wmnet
* 14:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2005-dev.codfw.wmnet
* 14:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet
* 14:53 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2004-dev.codfw.wmnet
* 14:53 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2007-dev.codfw.wmnet
* 14:51 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1054.eqiad.wmnet
* 14:51 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet
* 14:51 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet
* 14:47 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2007-dev.codfw.wmnet
* 14:47 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet
* 14:41 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2006-dev.codfw.wmnet
* 14:40 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet
* 14:37 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2005-dev.codfw.wmnet
* 14:37 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2005-dev.codfw.wmnet
* 14:29 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2005-dev.codfw.wmnet
* 14:29 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2006-dev.codfw.wmnet
* 14:21 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2006-dev.codfw.wmnet
* 14:21 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet
* 14:15 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2010-dev.codfw.wmnet
* 14:15 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2004-dev.codfw.wmnet
* 14:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1054.eqiad.wmnet with OS trixie
* 14:09 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudgw2004-dev.codfw.wmnet
* 14:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2003-dev.codfw.wmnet
* 14:02 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudgw2003-dev.codfw.wmnet
* 14:02 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet
* 13:53 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudlb2004-dev.codfw.wmnet
* 13:53 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2003-dev.codfw.wmnet
* 13:48 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 13:44 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudlb2003-dev.codfw.wmnet
* 13:44 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2002-dev.codfw.wmnet
* 13:42 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 13:41 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 13:41 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 13:37 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudlb2002-dev.codfw.wmnet
* 13:37 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudidp2001-dev.codfw.wmnet
* 13:33 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
* 13:33 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudidp2001-dev.codfw.wmnet
* 13:33 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
* 13:26 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudnet2006-dev.codfw.wmnet
* 13:26 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
* 13:23 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
* 13:18 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudnet2005-dev.codfw.wmnet
* 13:18 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2005-dev.codfw.wmnet
* 13:12 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudservices2005-dev.codfw.wmnet
* 13:11 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2004-dev.codfw.wmnet
* 13:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudservices2004-dev.codfw.wmnet
* 13:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudweb2002-dev.wikimedia.org
* 13:05 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:05 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1054
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1054
* 13:04 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1054
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1054.eqiad.wmnet 49.32.64.10.in-addr.arpa 9.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:04 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1054.eqiad.wmnet 49.32.64.10.in-addr.arpa 9.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:04 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1054 - blake@cumin1003&quot;
* 13:04 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1054 - blake@cumin1003&quot;
* 13:01 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudweb2002-dev.wikimedia.org
* 13:00 blake@cumin1003: START - Cookbook sre.dns.netbox
* 12:59 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1054
* 12:57 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1054.eqiad.wmnet with OS trixie
* 12:57 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet
* 12:56 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet
* 12:56 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1054.eqiad.wmnet
* 12:47 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS trixie
* 12:44 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:39 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 12:39 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 12:38 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 12:37 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 12:14 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:10 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:09 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:09 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:09 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:08 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 12:07 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:00 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 12:00 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
* 11:51 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 11:49 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
* 11:48 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:47 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:44 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:32 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet
* 11:32 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1053.eqiad.wmnet
* 11:32 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet
* 11:32 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet
* 11:32 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet
* 11:31 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2001.codfw.wmnet
* 11:31 cgoubert@cumin1003: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2001.codfw.wmnet
* 11:31 cgoubert@cumin1003: END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;wikikube-worker2001*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
* 11:30 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:30 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:21 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 18 hosts with reason: reboot &amp; upgrade
* 11:20 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS trixie
* 11:16 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:15 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:14 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 11:14 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 11 hosts
* 11:14 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 11 hosts
* 11:13 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 11:08 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 11:02 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1053.eqiad.wmnet with OS trixie
* 11:01 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 10:58 cgoubert@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 10:57 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:38 cgoubert@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS trixie
* 10:38 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
* 10:38 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
* 10:38 cgoubert@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;wikikube-worker2001*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
* 10:35 topranks: adjust IBGP outbound policy on lsw1-e2-codfw [[phab:T423430|T423430]] towards ssw1-e1-codfw
* 10:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 cgoubert@cumin1003: START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet
* 10:30 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:27 cgoubert@cumin1003: END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on A:wikikube-worker-codfw
* 10:27 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS bookworm
* 10:25 cgoubert@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 10:24 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 10:24 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 10:15 cgoubert@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage
* 10:11 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 10:11 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 10:08 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 10:08 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
* 10:07 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 10:06 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 10:00 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 09:55 cgoubert@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS bookworm
* 09:55 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2005-2006,2011-2012].codfw.wmnet
* 09:55 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
* 09:52 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2005-2006,2011-2012].codfw.wmnet
* 09:51 cgoubert@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on A:wikikube-worker-codfw
* 09:41 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage
* 09:37 topranks: apply new IBGP outbound policy on lsw1-e2-codfw [[phab:T423430|T423430]]
* 09:36 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:36 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage
* 09:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1053
* 09:16 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1053
* 09:15 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1053
* 09:15 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1053.eqiad.wmnet 48.32.64.10.in-addr.arpa 8.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:15 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1053.eqiad.wmnet 48.32.64.10.in-addr.arpa 8.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:15 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:15 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1053 - blake@cumin1003&quot;
* 09:15 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1053 - blake@cumin1003&quot;
* 09:11 blake@cumin1003: START - Cookbook sre.dns.netbox
* 09:11 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1053
* 09:08 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1053.eqiad.wmnet with OS trixie
* 09:08 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet
* 09:08 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet
* 09:08 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet
* 09:06 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1053.eqiad.wmnet
* 09:04 brouberol@dns1004: END - running authdns-update
* 09:03 brouberol@dns1004: START - running authdns-update
* 08:41 javiermonton@deploy2003: Finished deploy [analytics/refinery@1abf22e] (thin): Regular analytics weekly train THIN [analytics/refinery@1abf22ea] (duration: 02m 11s)
* 08:38 javiermonton@deploy2003: Started deploy [analytics/refinery@1abf22e] (thin): Regular analytics weekly train THIN [analytics/refinery@1abf22ea]
* 08:38 javiermonton@deploy2003: Finished deploy [analytics/refinery@1abf22e]: Regular analytics weekly train [analytics/refinery@1abf22ea] (duration: 05m 17s)
* 08:38 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:34 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 08:33 javiermonton@deploy2003: Started deploy [analytics/refinery@1abf22e]: Regular analytics weekly train [analytics/refinery@1abf22ea]
* 08:32 javiermonton@deploy2003: Finished deploy [analytics/refinery@1abf22e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1abf22ea] (duration: 02m 03s)
* 08:30 javiermonton@deploy2003: Started deploy [analytics/refinery@1abf22e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1abf22ea]
* 08:30 JavierMonton: Deploying Refinery at {{Gerrit|1abf22ea}} for changes 1308121/T427068 1306491/T430020 and {{Gerrit|1308190}}
* 08:29 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 08:29 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 08:24 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 08:24 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 08:18 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 08:18 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 08:00 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet
* 08:00 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet
* 07:52 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:52 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:49 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 11 hosts with reason: reboot &amp; upgrade
* 07:47 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:47 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:44 dcausse@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:44 dcausse@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:23 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 10 hosts
* 07:23 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 10 hosts
* 06:45 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 10 hosts with reason: reboot &amp; upgrade
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-09 ==
* 23:33 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308671{{!}}Enable section share on Persian Wikipedia (T431514)]] (duration: 13m 26s)
* 23:29 ladsgroup@deploy2003: ladsgroup, jdlrobson: Continuing with deployment
* 23:22 ladsgroup@deploy2003: ladsgroup, jdlrobson: Backport for [[gerrit:1308671{{!}}Enable section share on Persian Wikipedia (T431514)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:20 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1308671{{!}}Enable section share on Persian Wikipedia (T431514)]]
* 22:57 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1165.eqiad.wmnet
* 22:56 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1165.eqiad.wmnet
* 22:56 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1165.eqiad.wmnet
* 22:45 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1165.eqiad.wmnet with OS trixie
* 22:38 arlolra@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:37 arlolra@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:37 arlolra@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:37 arlolra@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:37 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 22:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1165.eqiad.wmnet with reason: host reimage
* 22:17 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1165.eqiad.wmnet with reason: host reimage
* 22:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 22:12 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 22:04 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 22:04 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1165
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1165
* 22:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1165
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1165.eqiad.wmnet 115.48.64.10.in-addr.arpa 5.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 22:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1165.eqiad.wmnet 115.48.64.10.in-addr.arpa 5.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1165 - jasmine@cumin2002&quot;
* 22:02 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1165 - jasmine@cumin2002&quot;
* 22:02 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 21:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox
* 21:55 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1165
* 21:54 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1165.eqiad.wmnet with OS trixie
* 21:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 21:54 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1165.eqiad.wmnet
* 21:53 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 21:53 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1165.eqiad.wmnet
* 21:53 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1165.eqiad.wmnet
* 21:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 21:47 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 21:45 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 21:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage
* 21:43 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 21:43 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 21:42 maryum: Deploy fix for [[phab:T431684|T431684]]
* 21:38 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage
* 21:27 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309290{{!}}Introduce sharing of section function (T18691)]], [[gerrit:1309291{{!}}Drop the share icon from the mobile site (T18691)]] (duration: 34m 14s)
* 21:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1002
* 21:23 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1002
* 21:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS trixie
* 21:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 21:20 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 22s)
* 21:20 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 21:16 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 21:16 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2001.codfw.wmnet -&gt; wcqs2002.codfw.wmnet, repooling source-only afterwards
* 21:15 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 21:13 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2002.codfw.wmnet with OS bookworm
* 21:11 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1309290{{!}}Introduce sharing of section function (T18691)]], [[gerrit:1309291{{!}}Drop the share icon from the mobile site (T18691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
* 21:07 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 6 hosts with reason: reboots
* 20:54 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecycle work - bking@cumin2003
* 20:53 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:53 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1309290{{!}}Introduce sharing of section function (T18691)]], [[gerrit:1309291{{!}}Drop the share icon from the mobile site (T18691)]]
* 20:51 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 20:47 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecycle work - bking@cumin2003
* 20:42 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 20:41 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:41 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 20:40 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host relforge1008.eqiad.wmnet
* 20:40 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:37 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1009.eqiad.wmnet with OS trixie
* 20:33 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:32 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:32 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: lifecyle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:31 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:31 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
* 20:29 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host relforge1008.eqiad.wmnet
* 20:24 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2002.codfw.wmnet with OS bookworm
* 20:23 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host relforge1008.eqiad.wmnet
* 20:23 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:23 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host relforge1008.eqiad.wmnet
* 20:22 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:22 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:21 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:21 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: Lifecycle work - bking@cumin2003 - [[phab:T431658|T431658]]
* 20:20 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1009.eqiad.wmnet with reason: host reimage
* 20:16 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 20:15 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1009.eqiad.wmnet with reason: host reimage
* 20:12 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 20:02 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views
* 19:55 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1009.eqiad.wmnet with OS trixie
* 19:54 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 19:43 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 19:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 19:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 19:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 19:25 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 18:42 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 18:41 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 18:16 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 18:15 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 17:45 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for doh5004.wikimedia.org
* 17:45 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for doh5004.wikimedia.org
* 17:38 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 17:35 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 17:29 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 17:26 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 17:09 mutante: zuul[12]00[123] - rebooting for maintenance
* 17:09 ebernhardson: start full in-place reindex of eqiad cirrussearch cluster
* 17:08 dzahn@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 17:08 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-cluster
* 17:03 ebernhardson: start full in-place reindex of codfw cirrussearch cluster
* 16:59 mutante: stewards1001/stewards2001 - reboot for maintenance
* 16:54 ebernhardson: start full in-place reindex of cloudelastic cluster
* 16:53 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 16:52 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 16:49 mutante: planet1003/planet2003 - rebooting
* 16:47 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on doh5004.wikimedia.org with reason: random high load, investigating
* 15:55 cjming@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 15:54 cjming@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 15:51 jynus: restarting backupmon1001
* 15:49 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 14 hosts
* 15:49 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 14 hosts
* 15:47 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backupmon1001.eqiad.wmnet with reason: restart
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:13 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:13 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 15:06 cjming@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 15:06 cjming@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 14:59 cjming@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
* 14:58 cjming@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
* 14:51 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 14 hosts
* 14:51 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 14 hosts
* 14:49 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 6 hosts with reason: reboot &amp; upgrade
* 14:48 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1002.eqiad.wmnet
* 14:48 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1002.eqiad.wmnet
* 14:47 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:45 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:42 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1052.eqiad.wmnet
* 14:42 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet
* 14:42 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet
* 14:32 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:32 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:31 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:31 elukey@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: sync
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:30 elukey@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: sync
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:30 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:28 elukey@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: sync
* 14:28 elukey@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: sync
* 14:26 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:23 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:23 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1052.eqiad.wmnet with OS trixie
* 14:19 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 6 hosts with reason: reboot &amp; upgrade
* 14:18 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:17 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:15 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:15 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 14:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:14 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:13 elukey: update druid indexation job for webrequest_sampled_live - [[phab:T427068|T427068]]
* 14:11 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:11 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:09 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating for papaul - jhancock@cumin2002&quot;
* 14:09 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating for papaul - jhancock@cumin2002&quot;
* 14:07 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:07 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply
* 14:04 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 14:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cuminunpriv1001.eqiad.wmnet
* 13:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1052.eqiad.wmnet with reason: host reimage
* 13:57 moritzm: installing requests security updates
* 13:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host cuminunpriv1001.eqiad.wmnet
* 13:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:53 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1052.eqiad.wmnet with reason: host reimage
* 13:50 moritzm: installing python-cryptography security updates
* 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 13:44 Msz2001: UTC afternoon config+backport window is done
* 13:44 Msz2001: Updated `logging` on `metawiki` to fix log performers, [[phab:T431176|T431176]]#12105297
* 13:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 13:43 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 13:43 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org
* 13:41 mszwarc@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309101{{!}}Fix UserEditTracker spoiling ActorStore cache for cross-wiki lookups (T431176)]], [[gerrit:1309122{{!}}SuggestedInvestigations: Instrument link clicks in the cases table (T429320)]] (duration: 07m 30s)
* 13:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org
* 13:37 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1052
* 13:36 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1052
* 13:35 mszwarc@deploy2003: mszwarc: Backport for [[gerrit:1309101{{!}}Fix UserEditTracker spoiling ActorStore cache for cross-wiki lookups (T431176)]], [[gerrit:1309122{{!}}SuggestedInvestigations: Instrument link clicks in the cases table (T429320)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:35 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1052
* 13:35 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1052.eqiad.wmnet 47.32.64.10.in-addr.arpa 7.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:35 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1052.eqiad.wmnet 47.32.64.10.in-addr.arpa 7.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:35 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:35 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1052 - blake@cumin1003&quot;
* 13:35 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1052 - blake@cumin1003&quot;
* 13:34 mszwarc@deploy2003: Started scap sync-world: Backport for [[gerrit:1309101{{!}}Fix UserEditTracker spoiling ActorStore cache for cross-wiki lookups (T431176)]], [[gerrit:1309122{{!}}SuggestedInvestigations: Instrument link clicks in the cases table (T429320)]]
* 13:31 blake@cumin1003: START - Cookbook sre.dns.netbox
* 13:31 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1052
* 13:30 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1052.eqiad.wmnet with OS trixie
* 13:30 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet
* 13:29 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet
* 13:29 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1052.eqiad.wmnet
* 13:17 jforrester@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309112{{!}}abstractwiki: Don't run our jobs too quickly, they'll collide and error (T430898)]] (duration: 11m 26s)
* 13:13 jforrester@deploy2003: jforrester: Continuing with deployment
* 13:08 jforrester@deploy2003: jforrester: Backport for [[gerrit:1309112{{!}}abstractwiki: Don't run our jobs too quickly, they'll collide and error (T430898)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:06 jforrester@deploy2003: Started scap sync-world: Backport for [[gerrit:1309112{{!}}abstractwiki: Don't run our jobs too quickly, they'll collide and error (T430898)]]
* 12:54 cgoubert@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 12:54 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:52 cgoubert@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 12:45 cgoubert@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 12:44 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 12:44 cgoubert@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 12:43 cgoubert@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 12:43 cgoubert@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 12:42 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org
* 12:20 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org
* 12:20 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org
* 12:18 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast4006.wikimedia.org
* 12:18 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 12:18 cmooney@cumin1003: END (PASS) - Cookbook sre.network.host-bgp (exit_code=0) for host ml-serve1004
* 12:18 cmooney@cumin1003: START - Cookbook sre.network.host-bgp for host ml-serve1004
* 12:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org
* 12:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org
* 12:10 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup[2003,2014].codfw.wmnet with reason: reboot &amp; upgrade
* 12:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet
* 12:09 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1003.eqiad.wmnet
* 12:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet
* 12:05 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid1003.eqiad.wmnet
* 12:03 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup[1003,1014].eqiad.wmnet with reason: reboot &amp; upgrade
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2003.codfw.wmnet
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 11:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid2003.codfw.wmnet
* 11:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 11:55 jmm@dns1004: END - running authdns-update
* 11:53 jmm@dns1004: START - running authdns-update
* 11:50 jmm@dns1004: END - running authdns-update
* 11:48 jmm@dns1004: START - running authdns-update
* 11:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org
* 11:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org
* 11:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet
* 11:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet
* 11:12 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet
* 11:12 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet
* 11:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org
* 11:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org
* 11:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org
* 11:04 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org
* 11:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 11:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2005.codfw.wmnet
* 11:00 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:00 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 10:59 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 10:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 10:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 10:55 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 10:51 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 10:51 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:50 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2005.codfw.wmnet
* 10:41 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:40 cmooney@cumin1003: END (PASS) - Cookbook sre.network.host-bgp (exit_code=0) for host ml-serve1003
* 10:40 cmooney@cumin1003: START - Cookbook sre.network.host-bgp for host ml-serve1003
* 10:39 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet
* 10:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org
* 10:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org
* 10:35 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1004.eqiad.wmnet with OS bookworm
* 10:31 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org
* 10:31 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org
* 10:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org
* 10:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org
* 10:29 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - [[phab:T431656|T431656]]
* 10:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org
* 10:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org
* 10:23 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet
* 10:21 moritzm: failover Ganeti master in codfw/routed to ganeti2034 [[phab:T430928|T430928]]
* 10:19 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2031.codfw.wmnet to cluster codfw and group B
* 10:19 moritzm: readded ganeti2031 to the codfw Ganeti cluster [[phab:T430910|T430910]]
* 10:18 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1004.eqiad.wmnet with reason: host reimage
* 10:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org
* 10:18 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1003
* 10:18 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1003
* 10:17 jmm@cumin2003: START - Cookbook sre.ganeti.addnode for new host ganeti2031.codfw.wmnet to cluster codfw and group B
* 10:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org
* 10:16 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org
* 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org
* 10:15 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1004.eqiad.wmnet with reason: host reimage
* 10:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1001.eqiad.wmnet
* 10:14 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 10:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
* 10:00 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ml-serve1004
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1004
* 09:57 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
* 09:57 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1004
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1004.eqiad.wmnet 50.48.64.10.in-addr.arpa 0.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:57 klausman@cumin1003: START - Cookbook sre.dns.wipe-cache ml-serve1004.eqiad.wmnet 50.48.64.10.in-addr.arpa 0.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:57 klausman@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1004 - klausman@cumin1003&quot;
* 09:56 klausman@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1004 - klausman@cumin1003&quot;
* 09:55 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-coord1001.eqiad.wmnet
* 09:55 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 09:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet
* 09:52 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-coord1001.eqiad.wmnet
* 09:52 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 09:52 klausman@cumin1003: START - Cookbook sre.dns.netbox
* 09:50 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
* 09:50 klausman@cumin1003: START - Cookbook sre.hosts.move-vlan for host ml-serve1004
* 09:50 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1004.eqiad.wmnet with OS bookworm
* 09:50 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1003.eqiad.wmnet with OS bookworm
* 09:50 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1001.eqiad.wmnet
* 09:49 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet
* 09:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2007.codfw.wmnet
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 09:49 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet
* 09:49 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: testvm2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 09:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet
* 09:39 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet
* 09:39 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 09:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet
* 09:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet
* 09:35 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet
* 09:34 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2007.codfw.wmnet
* 09:33 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet
* 09:33 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet
* 09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet
* 09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:27 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 09:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet
* 09:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet
* 09:23 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309102{{!}}NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] (duration: 06m 27s)
* 09:23 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2205: codfw rack B4 repool after maintenance
* 09:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:23 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2204: codfw rack B4 repool after maintenance
* 09:19 urbanecm@deploy2003: urbanecm: Continuing with deployment
* 09:19 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1309102{{!}}NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:18 jmm@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: reboot
* 09:17 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1309102{{!}}NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]]
* 09:08 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ml-serve1003
* 09:08 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1003
* 09:07 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1003
* 09:07 klausman@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1003.eqiad.wmnet 81.32.64.10.in-addr.arpa 1.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:07 klausman@cumin1003: START - Cookbook sre.dns.wipe-cache ml-serve1003.eqiad.wmnet 81.32.64.10.in-addr.arpa 1.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:07 klausman@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:07 klausman@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1003 - klausman@cumin1003&quot;
* 09:06 klausman@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1003 - klausman@cumin1003&quot;
* 08:58 klausman@cumin1003: START - Cookbook sre.dns.netbox
* 08:57 klausman@cumin1003: START - Cookbook sre.hosts.move-vlan for host ml-serve1003
* 08:57 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1003.eqiad.wmnet with OS bookworm
* 08:55 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve1003.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 08:55 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet
* 08:55 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet
* 08:55 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1003.eqiad.wmnet with OS bookworm
* 08:39 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 08:38 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2205: codfw rack B4 repool after maintenance
* 08:37 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2204: codfw rack B4 repool after maintenance
* 08:36 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage
* 08:35 hashar@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 08:32 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:32 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:31 hashar@deploy2003: Rolling back deployment
* 08:26 moritzm: failover Ganeti master in codfw to ganeti2048 [[phab:T430928|T430928]]
* 08:16 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1003.eqiad.wmnet with OS bookworm
* 08:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2004.codfw.wmnet
* 08:16 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet
* 08:16 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet
* 08:16 klausman@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve1003.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 08:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet
* 08:15 XioNoX: lsw1-b4-codfw&gt; request system reboot - [[phab:T430910|T430910]]
* 08:15 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b4-codfw,lsw1-b4-codfw IPv6,lsw1-b4-codfw.mgmt with reason: Switch maintenance
* 08:13 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for codfw rack B4
* 08:13 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2004.codfw.wmnet
* 08:10 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet
* 08:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet
* 08:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2205: codfw rack B4 depool for maintenance
* 08:08 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2205: codfw rack B4 depool for maintenance
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2003.codfw.wmnet
* 08:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2204: codfw rack B4 depool for maintenance
* 08:08 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2204: codfw rack B4 depool for maintenance
* 08:08 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 27 hosts with reason: codfw rack B4 depool for maintenance
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cumin2003.codfw.wmnet
* 07:56 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack B4
* 07:54 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1008.eqiad.wmnet with OS trixie
* 07:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308687{{!}}Enable sub-references on more group2 wikis (batch2) (T430941)]] (duration: 08m 36s)
* 07:44 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 07:43 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1308687{{!}}Enable sub-references on more group2 wikis (batch2) (T430941)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:41 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1308687{{!}}Enable sub-references on more group2 wikis (batch2) (T430941)]]
* 07:35 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1008.eqiad.wmnet with reason: host reimage
* 07:31 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1008.eqiad.wmnet with reason: host reimage
* 07:12 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1008.eqiad.wmnet with OS trixie
* 07:10 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 07:00 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 06:59 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1008.eqiad.wmnet with OS trixie
* 06:57 Emperor: rebalance thanos swift rings after previous re-image of thanos-fe1004 to trixie
* 06:47 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1008.eqiad.wmnet with OS trixie
* 04:10 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cp6008.drmrs.wmnet with reason: Hardware failure - [[phab:T431651|T431651]]
* 03:55 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6008.*
* 03:29 ryankemper: [[phab:T431311|T431311]] Repooled eqiad cirrussearch clusters (`chi/omega/psi`) following completion of OpenSearch 2.19 migration
* 03:27 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=eqiad
* 03:27 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=eqiad
* 03:27 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image

== 2026-07-08 ==
* 23:52 Amir1: ladsgroup@deploy2003:~$ mwscript-k8s --follow -- extensions/ORES/maintenance/PurgeScoreCache.php --wiki=simplewiki --model damaging --old ([[phab:T431159|T431159]])
* 23:46 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 23:46 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add missing PTR for 2001:df2:e500:fe08::1 - cmooney@cumin1003&quot;
* 23:46 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add missing PTR for 2001:df2:e500:fe08::1 - cmooney@cumin1003&quot;
* 23:40 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 23:16 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 23:15 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 22:42 rzl@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 22:40 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308782{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] (duration: 12m 55s)
* 22:40 rzl@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 22:37 rzl@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 22:36 rzl@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 22:35 rzl@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 22:34 urbanecm@deploy2003: urbanecm: Continuing with deployment
* 22:33 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1308782{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 rzl@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 22:32 rzl@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 22:30 rzl@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 22:30 rzl@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 22:29 rzl@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 22:27 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1308782{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781{{!}}fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]]
* 22:26 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 22:22 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 22:21 rzl@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 22:19 rzl@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 22:19 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 22:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 22:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 22:13 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage
* 22:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 22:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 22:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 22:06 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 22:05 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1094.eqiad.wmnet with OS trixie
* 22:01 urbanecm: Make https://test.wikipedia.org/w/index.php?title=MediaWiki:GrowthExperimentsSuggestedEdits.json&amp;diff=prev&amp;oldid=750552 with GrowthExperiments disabled (via mw-experimental), then run `\MediaWiki\MediaWikiServices::getInstance()-&gt;get('CommunityConfiguration.ProviderFactory')-&gt;newProvider('GrowthSuggestedEdits')-&gt;getStore()-&gt;invalidate()` ([[phab:T431625|T431625]])
* 21:56 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d2-codfw
* 21:55 urbanecm@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 21:55 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d2-codfw
* 21:55 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c4-codfw
* 21:55 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c4-codfw
* 21:55 urbanecm@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2002
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2002
* 21:54 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2002
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2002.codfw.wmnet 50.32.192.10.in-addr.arpa 0.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:54 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2002.codfw.wmnet 50.32.192.10.in-addr.arpa 0.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:54 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2002 - bking@cumin2003&quot;
* 21:54 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2002 - bking@cumin2003&quot;
* 21:49 bking@cumin2003: START - Cookbook sre.dns.netbox
* 21:49 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2002
* 21:49 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2002.codfw.wmnet with OS trixie
* 21:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1094.eqiad.wmnet with reason: host reimage
* 21:42 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 21:39 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 21:37 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1094.eqiad.wmnet with reason: host reimage
* 21:36 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 21:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 21:29 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 21:27 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 21:22 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1094.eqiad.wmnet with OS trixie
* 21:21 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2039.codfw.wmnet with OS bullseye
* 21:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 21:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 21:04 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2039.codfw.wmnet with reason: host reimage
* 21:00 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2039.codfw.wmnet with reason: host reimage
* 20:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1073.eqiad.wmnet with OS trixie
* 20:48 mutante: deploy2003 - kill 1102 (stunnel4) ; systemctl start stunnel4 ([[phab:T418262|T418262]])
* 20:42 cjming@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308183{{!}}Move Test Kitchen config from CommonSettings.php (T431257)]] (duration: 33m 02s)
* 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2039.codfw.wmnet with OS bullseye
* 20:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1073.eqiad.wmnet with reason: host reimage
* 20:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1073.eqiad.wmnet with reason: host reimage
* 20:30 cjming@deploy2003: cjming: Continuing with deployment
* 20:28 cjming@deploy2003: cjming: Backport for [[gerrit:1308183{{!}}Move Test Kitchen config from CommonSettings.php (T431257)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:20 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1098.eqiad.wmnet with OS trixie
* 20:13 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1073.eqiad.wmnet with OS trixie
* 20:09 cjming@deploy2003: Started scap sync-world: Backport for [[gerrit:1308183{{!}}Move Test Kitchen config from CommonSettings.php (T431257)]]
* 20:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1098.eqiad.wmnet with reason: host reimage
* 19:56 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1098.eqiad.wmnet with reason: host reimage
* 19:55 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d4-codfw
* 19:54 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d4-codfw
* 19:54 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-codfw
* 19:54 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c1-codfw
* 19:52 mutante: restarting gerrit on gerrit.wikimedia.org (gerrit2003)
* 19:48 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2331.codfw.wmnet
* 19:48 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2331.codfw.wmnet
* 19:48 mutante: restarting gerrit on gerrit-replica.wikimedia.org (gerrit1003)
* 19:46 mutante: restarting gerrit on gerrit-spare.wikimedia.org (gerrit2002)
* 19:43 jasmine@cumin2002: conftool action : set/pooled=yes; selector: name=wikikube-worker2331.codfw.wmnet,cluster=kubernetes,service=kubesvc
* 19:43 jasmine@cumin2002: conftool action : set/weight=10; selector: name=wikikube-worker2331.codfw.wmnet,cluster=kubernetes,service=kubesvc
* 19:40 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1098.eqiad.wmnet with OS trixie
* 19:38 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d5-codfw
* 19:38 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d5-codfw
* 19:38 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c7-codfw
* 19:38 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c7-codfw
* 19:38 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c5-codfw
* 19:38 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c5-codfw
* 19:30 jasmine_: ran homer on lsw1-d8-codfw, adding wikikube-worker2331 to cluster
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1100.eqiad.wmnet with OS trixie
* 19:20 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d8-codfw
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d8-codfw
* 19:19 mutante: gerrit - replacing private key for registerEmail verification
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-magru
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device cr2-magru
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d7-codfw
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d7-codfw
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw
* 19:19 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d3-codfw
* 19:19 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d1-codfw
* 19:18 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d1-codfw
* 19:18 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c2-codfw
* 19:18 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c2-codfw
* 19:11 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-codfw
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d1-codfw
* 19:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-magru
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device cr1-magru
* 19:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d8-codfw
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d8-codfw
* 19:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-codfw
* 19:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1100.eqiad.wmnet with reason: host reimage
* 19:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d6-codfw
* 19:09 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c6-codfw
* 19:09 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c6-codfw
* 19:09 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c3-codfw
* 19:09 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c3-codfw
* 19:09 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b4-magru
* 19:09 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-b4-magru
* 19:08 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b3-magru
* 19:08 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-b3-magru
* 19:05 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1100.eqiad.wmnet with reason: host reimage
* 19:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1122.eqiad.wmnet with OS trixie
* 19:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 18:59 topranks: rolling out update to BGP ACL on Nokia Switches eqiad, codfw &amp; ulsfo [[phab:T425703|T425703]]
* 18:58 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 18:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 18:55 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 18:53 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 18:52 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 18:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1100.eqiad.wmnet with OS trixie
* 18:48 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1068.eqiad.wmnet with OS trixie
* 18:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1102.eqiad.wmnet with OS trixie
* 18:47 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 18:46 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 18:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1122.eqiad.wmnet with reason: host reimage
* 18:38 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1122.eqiad.wmnet with reason: host reimage
* 18:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1068.eqiad.wmnet with reason: host reimage
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1122
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1122
* 18:25 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1122
* 18:25 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1122.eqiad.wmnet 31.48.64.10.in-addr.arpa 1.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:25 bking@cumin2003: START - Cookbook sre.dns.wipe-cache cirrussearch1122.eqiad.wmnet 31.48.64.10.in-addr.arpa 1.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:25 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:25 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1122 - bking@cumin2003&quot;
* 18:25 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1122 - bking@cumin2003&quot;
* 18:21 rzl@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 18:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1068.eqiad.wmnet with reason: host reimage
* 18:21 rzl@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 18:21 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 18:20 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 18:19 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 18:19 bking@cumin2003: START - Cookbook sre.dns.netbox
* 18:18 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1122
* 18:18 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1122.eqiad.wmnet with OS trixie
* 18:15 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 18:15 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 18:13 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 18:13 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 18:10 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 18:05 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1068.eqiad.wmnet with OS trixie
* 18:01 kamila@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - [[phab:T423714|T423714]] (duration: 18m 29s)
* 18:00 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:55 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:46 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:42 kamila@deploy2003: Started scap sync-world: Test deployment to validate deployment server switchover - [[phab:T423714|T423714]]
* 17:42 kamila@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - [[phab:T423714|T423714]] (duration: 19m 50s)
* 17:42 kamila@deploy2003: Rolling back deployment
* 17:35 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1102.eqiad.wmnet with OS trixie
* 17:31 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 17:18 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 17:16 kamila@deploy1003: Unlocked for deployment [MediaWiki]: switching deployment server (duration: 22m 07s)
* 17:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 17:11 kamila@dns1005: END - running authdns-update
* 17:09 kamila@dns1005: START - running authdns-update
* 17:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage
* 17:04 jasmine@cumin2002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1164.eqiad.wmnet
* 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1164.eqiad.wmnet
* 17:04 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1164.eqiad.wmnet
* 16:56 kamila@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet,releases1003.eqiad.wmnet with reason: Deployment server switchover
* 16:54 kamila@deploy1003: Locking from deployment [MediaWiki]: switching deployment server
* 16:53 kamila@deploy1003: Unlocked for deployment [MediaWiki]: switching deployment server (duration: 04m 02s)
* 16:52 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie
* 16:49 kamila@deploy1003: Locking from deployment [MediaWiki]: switching deployment server
* 16:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1095.eqiad.wmnet with OS trixie
* 16:45 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1093.eqiad.wmnet with OS trixie
* 16:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1164.eqiad.wmnet with OS trixie
* 16:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1095.eqiad.wmnet with reason: host reimage
* 16:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 16:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 16:23 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage
* 16:18 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cirrussearch1093.eqiad.wmnet with reason: host reimage
* 16:16 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage
* 16:15 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1095.eqiad.wmnet with reason: host reimage
* 16:09 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 16:09 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 16:08 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1093.eqiad.wmnet with reason: host reimage
* 15:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Pool test
* 15:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:59 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Pool test
* 15:58 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Depool test
* 15:58 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:58 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:58 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Depool test
* 15:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1164
* 15:57 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1164
* 15:57 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1164
* 15:56 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1164.eqiad.wmnet 114.48.64.10.in-addr.arpa 4.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:56 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1164.eqiad.wmnet 114.48.64.10.in-addr.arpa 4.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:56 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:56 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1164 - jasmine@cumin2002&quot;
* 15:56 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1164 - jasmine@cumin2002&quot;
* 15:52 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1093.eqiad.wmnet with OS trixie
* 15:51 jasmine@cumin2002: START - Cookbook sre.dns.netbox
* 15:51 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1164
* 15:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1164.eqiad.wmnet with OS trixie
* 15:50 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet
* 15:50 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet
* 15:50 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1164.eqiad.wmnet
* 15:48 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1095.eqiad.wmnet with OS trixie
* 15:42 jasmine@cumin2002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1164.eqiad.wmnet
* 15:42 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet
* 15:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:42 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet
* 15:42 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1164.eqiad.wmnet
* 15:39 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 15:39 elukey@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 15:36 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1007.eqiad.wmnet with OS trixie
* 15:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Pool test
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:19 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage
* 15:15 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
* 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet
* 15:15 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
* 15:15 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 15:15 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage
* 15:15 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:14 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Pool test
* 15:14 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet
* 15:14 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: Depool test
* 15:14 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2228: Depool test
* 15:10 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 15:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 15:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 15:08 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 15:06 blake@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet
* 15:06 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 15:06 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 15:05 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet
* 15:05 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 15:04 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:04 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 15:04 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:03 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 15:03 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 15:03 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:03 swfrench-wmf: restarted navtiming on webperf2003 - [[phab:T430909|T430909]]
* 15:03 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 15:03 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 15:03 swfrench-wmf: restarted eqsin, codfw confds - [[phab:T430909|T430909]]
* 15:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test1002.eqiad.wmnet
* 15:02 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet
* 14:59 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:59 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet
* 14:55 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS trixie
* 14:52 swfrench-wmf: restarted ulsfo confds, confirmed now connected to codfw backends except those using wikimedia.org SRV record - [[phab:T430909|T430909]]
* 14:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:50 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:49 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 14:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:41 moritzm: uninstalling dhcpcd-base from trixie hosts which still have it installed [[phab:T414341|T414341]]
* 14:40 sukhe: sudo cumin -b1 -s120 &quot;P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2011*&lt;nowiki&gt;}&lt;/nowiki&gt; or P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2012*&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; &quot;systemctl restart pybal.service&quot;
* 14:39 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 14:39 mvernon@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host thanos-be1007.eqiad.wmnet with OS trixie
* 14:37 sukhe: restart pybal on lvs2013 to revert back to conf2004
* 14:35 sukhe: restart pybal on lvs2014 to revert back to conf2004
* 14:34 swfrench-wmf: switched codfw, eqsin, ulsfo etcd client SRV records back to codfw - [[phab:T430909|T430909]]
* 14:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1002.eqiad.wmnet
* 14:32 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet
* 14:31 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS trixie
* 14:31 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:31 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:31 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:30 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:30 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Pool test
* 14:30 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:29 swfrench@dns1004: END - running authdns-update
* 14:29 moritzm: installing jackson-core security updates
* 14:27 swfrench@dns1004: START - running authdns-update
* 14:22 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:22 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:22 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1119.eqiad.wmnet with OS trixie
* 14:22 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:21 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:20 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:20 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 14:20 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:19 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 14:19 moritzm: installing librabbitmq security updates
* 14:19 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1002.eqiad.wmnet
* 14:18 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet
* 14:16 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:16 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:16 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:15 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:15 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:14 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Pool test
* 14:14 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 14:14 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet
* 14:10 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 14:08 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet
* 14:05 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1118.eqiad.wmnet with OS trixie
* 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test1001.eqiad.wmnet
* 14:00 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-worker@eqiad
* 14:00 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 13:59 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 13:57 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1119.eqiad.wmnet with reason: host reimage
* 13:54 moritzm: installing libcap2 security updates
* 13:53 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1119.eqiad.wmnet with reason: host reimage
* 13:52 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet
* 13:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0)
* 13:52 fceratto@cumin1003: START - Cookbook sre.mysql.depool
* 13:50 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-worker@eqiad
* 13:50 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1051.eqiad.wmnet
* 13:50 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet
* 13:50 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet
* 13:49 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 13:45 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1081.eqiad.wmnet with OS trixie
* 13:41 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1119
* 13:41 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1119
* 13:40 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1119
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1119.eqiad.wmnet 97.32.64.10.in-addr.arpa 7.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:40 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1119.eqiad.wmnet 97.32.64.10.in-addr.arpa 7.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1119 - atsuko@cumin1003&quot;
* 13:40 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1119 - atsuko@cumin1003&quot;
* 13:40 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1118.eqiad.wmnet with reason: host reimage
* 13:39 moritzm: installing krb5 security updates
* 13:37 Lucas_WMDE: UTC afternoon backport+config window done
* 13:37 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1006.eqiad.wmnet with OS trixie
* 13:36 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1118.eqiad.wmnet with reason: host reimage
* 13:36 atsuko@cumin1003: START - Cookbook sre.dns.netbox
* 13:35 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308622{{!}}Enable Article Guidance extension on itwiki (T431540)]] (duration: 07m 46s)
* 13:34 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1119
* 13:34 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1119.eqiad.wmnet with OS trixie
* 13:30 sbisson@deploy1003: sbisson: Continuing with deployment
* 13:30 moritzm: installing openssh security updates
* 13:30 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
* 13:29 sbisson@deploy1003: sbisson: Backport for [[gerrit:1308622{{!}}Enable Article Guidance extension on itwiki (T431540)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 13:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 13:27 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1308622{{!}}Enable Article Guidance extension on itwiki (T431540)]]
* 13:26 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1051.eqiad.wmnet with OS trixie
* 13:24 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage
* 13:24 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1118
* 13:24 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1118
* 13:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308553{{!}}Deploy IRS to enwiki (T431316)]] (duration: 12m 12s)
* 13:21 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage
* 13:18 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage
* 13:18 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1118
* 13:18 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1118.eqiad.wmnet 90.32.64.10.in-addr.arpa 0.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:18 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1118.eqiad.wmnet 90.32.64.10.in-addr.arpa 0.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:18 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:18 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1118 - atsuko@cumin1003&quot;
* 13:18 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1118 - atsuko@cumin1003&quot;
* 13:17 stran@deploy1003: stran: Continuing with deployment
* 13:16 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:15 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:13 atsuko@cumin1003: START - Cookbook sre.dns.netbox
* 13:12 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1118
* 13:12 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage
* 13:12 stran@deploy1003: stran: Backport for [[gerrit:1308553{{!}}Deploy IRS to enwiki (T431316)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:12 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1118.eqiad.wmnet with OS trixie
* 13:10 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1308553{{!}}Deploy IRS to enwiki (T431316)]]
* 13:05 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-worker@codfw
* 13:05 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 13:05 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1081.eqiad.wmnet with OS trixie
* 13:05 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage
* 13:04 moritzm: installing jq security updates
* 13:04 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 13:01 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage
* 12:58 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-worker@codfw
* 12:52 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:50 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1051
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1051
* 12:43 moritzm: installing Python 3.11 security updates
* 12:43 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1051
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1051.eqiad.wmnet 46.32.64.10.in-addr.arpa 6.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:43 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1051.eqiad.wmnet 46.32.64.10.in-addr.arpa 6.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:43 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1051 - blake@cumin1003&quot;
* 12:43 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1051 - blake@cumin1003&quot;
* 12:38 blake@cumin1003: START - Cookbook sre.dns.netbox
* 12:38 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1051
* 12:38 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1051.eqiad.wmnet with OS trixie
* 12:37 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet
* 12:36 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet
* 12:36 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1051.eqiad.wmnet
* 12:34 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:34 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:27 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:27 mvernon@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 12:02 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 12:01 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1006.eqiad.wmnet with OS trixie
* 11:43 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie
* 11:38 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 11:32 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1076.eqiad.wmnet with OS trixie
* 11:26 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1075.eqiad.wmnet with OS trixie
* 11:21 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet
* 11:19 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet
* 11:19 moritzm: temporarily remove ganeti2031 from codfw cluster [[phab:T430910|T430910]]
* 11:08 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1076.eqiad.wmnet with reason: host reimage
* 11:08 moritzm: installing Linux 6.1.176 on Bookworm servers
* 11:03 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1076.eqiad.wmnet with reason: host reimage
* 11:00 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1075.eqiad.wmnet with reason: host reimage
* 10:56 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1075.eqiad.wmnet with reason: host reimage
* 10:47 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1076.eqiad.wmnet with OS trixie
* 10:46 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1074.eqiad.wmnet with OS trixie
* 10:45 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1005.eqiad.wmnet with OS trixie
* 10:40 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1075.eqiad.wmnet with OS trixie
* 10:32 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 10:29 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage
* 10:25 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage
* 10:24 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1074.eqiad.wmnet with reason: host reimage
* 10:17 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1074.eqiad.wmnet with reason: host reimage
* 10:12 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1005.eqiad.wmnet with OS trixie
* 10:12 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet
* 10:04 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 10:01 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet
* 10:01 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1074.eqiad.wmnet with OS trixie
* 10:01 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet
* 09:43 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 09:43 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
* 09:43 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 09:35 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
* 09:34 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 09:34 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 09:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 41 days, 15:00:00 on db2252.codfw.wmnet with reason: Test
* 09:32 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 09:32 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 09:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: codfw rack B3 pool after maintenance
* 09:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 09:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 09:07 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 09:02 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0)
* 08:57 topranks: merge patch to shift eqiad &lt;-&gt; esams traffic onto new 40G circuit
* 08:54 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 08:50 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1004.eqiad.wmnet with OS trixie
* 08:50 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart
* 08:50 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99)
* 08:50 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart
* 08:45 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool es2051: codfw rack B3 pool after maintenance
* 08:44 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:44 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:43 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet
* 08:43 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet
* 08:42 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 08:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 08:40 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 08:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:35 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis minwikiquote in section s3
* 08:33 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s3
* 08:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5
* 08:30 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5
* 08:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis minwikiquote in section s5
* 08:27 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage
* 08:23 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage
* 08:23 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:22 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s5
* 08:19 XioNoX: lsw1-b3-codfw&gt; request system reboot - [[phab:T430909|T430909]]
* 08:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5
* 08:17 hashar@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 08:16 cwilliams@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5
* 08:15 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for codfw rack B3
* 08:15 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet
* 08:15 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-b3-codfw,lsw1-b3-codfw IPv6,lsw1-b3-codfw.mgmt with reason: Switch maintenance
* 08:15 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet
* 08:13 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet
* 08:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: codfw rack B3 depool for maintenance
* 08:05 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool es2051: codfw rack B3 depool for maintenance
* 08:05 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1004.eqiad.wmnet with OS trixie
* 08:03 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1005.eqiad.wmnet with OS trixie
* 08:03 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 21 hosts with reason: codfw rack B3 depool for maintenance
* 07:56 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack B3
* 07:44 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1005.eqiad.wmnet with reason: host reimage
* 07:38 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1005.eqiad.wmnet with reason: host reimage
* 07:36 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet with OS bookworm
* 07:29 moritzm: installing gnutls28 security updates
* 07:21 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1005.eqiad.wmnet with OS trixie
* 07:13 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1125.eqiad.wmnet with OS trixie
* 07:13 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1005.eqiad.wmnet with reason: host reimage
* 07:07 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1005.eqiad.wmnet with reason: host reimage
* 06:56 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1005.eqiad.wmnet with OS bookworm
* 06:54 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1125.eqiad.wmnet with reason: host reimage
* 06:52 elukey: upgrade all trixie hosts to pywmflib 3.1 - [[phab:T430552|T430552]]
* 06:50 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1125.eqiad.wmnet with reason: host reimage
* 06:40 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 06:40 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 06:38 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1125.eqiad.wmnet with OS trixie
* 05:42 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1107.eqiad.wmnet with OS trixie
* 05:35 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1124.eqiad.wmnet with OS trixie
* 05:31 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1101.eqiad.wmnet with OS trixie
* 05:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1107.eqiad.wmnet with reason: host reimage
* 05:17 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1124.eqiad.wmnet with reason: host reimage
* 05:13 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1107.eqiad.wmnet with reason: host reimage
* 05:13 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1101.eqiad.wmnet with reason: host reimage
* 05:11 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1124.eqiad.wmnet with reason: host reimage
* 05:10 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1101.eqiad.wmnet with reason: host reimage
* 04:58 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1124.eqiad.wmnet with OS trixie
* 04:56 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1107.eqiad.wmnet with OS trixie
* 04:55 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1101.eqiad.wmnet with OS trixie
* 02:27 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308251{{!}}Activate minwikiquote (T429922)]] (duration: 08m 14s)
* 02:22 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 02:21 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1308251{{!}}Activate minwikiquote (T429922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 02:19 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1308251{{!}}Activate minwikiquote (T429922)]]
* 01:59 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308250{{!}}Init minwikiquote (T429922)]] (duration: 09m 46s)
* 01:55 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 01:51 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1308250{{!}}Init minwikiquote (T429922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 01:49 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1308250{{!}}Init minwikiquote (T429922)]]
* 01:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1099.eqiad.wmnet with OS trixie
* 00:57 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1110.eqiad.wmnet with OS trixie
* 00:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1099.eqiad.wmnet with reason: host reimage
* 00:41 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1099.eqiad.wmnet with reason: host reimage
* 00:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1110.eqiad.wmnet with reason: host reimage
* 00:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1110.eqiad.wmnet with reason: host reimage
* 00:26 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1099.eqiad.wmnet with OS trixie
* 00:15 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1110.eqiad.wmnet with OS trixie

== 2026-07-07 ==
* 22:49 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1097.eqiad.wmnet with OS trixie
* 22:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1097.eqiad.wmnet with reason: host reimage
* 22:24 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1097.eqiad.wmnet with reason: host reimage
* 22:14 hashar: Restarting Gerrit on gerrit2002 and gerrit1003 (replicas)
* 22:09 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1097.eqiad.wmnet with OS trixie
* 22:07 hashar: Restarting Gerrit on gerrit2003
* 21:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:11 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 21:05 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1108.eqiad.wmnet with OS trixie
* 20:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1091.eqiad.wmnet with OS trixie
* 20:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1090.eqiad.wmnet with OS trixie
* 20:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1108.eqiad.wmnet with reason: host reimage
* 20:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1091.eqiad.wmnet with reason: host reimage
* 20:33 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1090.eqiad.wmnet with reason: host reimage
* 20:33 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1091.eqiad.wmnet with reason: host reimage
* 20:30 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1108.eqiad.wmnet with reason: host reimage
* 20:30 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 20:30 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1090.eqiad.wmnet with reason: host reimage
* 20:30 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 20:27 jasmine_: &quot;homer lsw1-c2-eqiad* commit &quot;Added new stacked control plane wikikube-ctrl1006&quot;&quot;
* 20:22 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308196{{!}}Bump Parsoid image limit to 1250 (T430854)]] (duration: 07m 29s)
* 20:20 jasmine_: &quot;homer &quot;cr*eqiad*&quot; commit &quot;Added new stacked control plane wikikube-ctrl1006&quot;&quot;
* 20:18 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1091.eqiad.wmnet with OS trixie
* 20:17 arlolra@deploy1003: arlolra: Continuing with deployment
* 20:16 arlolra@deploy1003: arlolra: Backport for [[gerrit:1308196{{!}}Bump Parsoid image limit to 1250 (T430854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:16 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1090.eqiad.wmnet with OS trixie
* 20:15 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1108.eqiad.wmnet with OS trixie
* 20:14 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1308196{{!}}Bump Parsoid image limit to 1250 (T430854)]]
* 20:09 cwhite: remove 2026-04 swift log archives from centrallog2002 to free some space
* 20:01 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cirrussearch1108.eqiad.wmnet with OS trixie
* 19:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1108.eqiad.wmnet with OS trixie
* 19:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1090.eqiad.wmnet with OS trixie
* 19:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1109.eqiad.wmnet with OS trixie
* 19:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1092.eqiad.wmnet with OS trixie
* 19:32 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1123.eqiad.wmnet with OS trixie
* 19:22 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1109.eqiad.wmnet with reason: host reimage
* 19:19 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=dns7002.*
* 19:17 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1092.eqiad.wmnet with reason: host reimage
* 19:17 jasmine@dns1004: END - running authdns-update
* 19:15 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1109.eqiad.wmnet with reason: host reimage
* 19:15 jasmine@dns1004: START - running authdns-update
* 19:15 cdobbins@dns1004: END - running authdns-update
* 19:14 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1123.eqiad.wmnet with reason: host reimage
* 19:13 cdobbins@dns1004: START - running authdns-update
* 19:12 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=dns7002.*,service=authdns-update
* 19:11 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1092.eqiad.wmnet with reason: host reimage
* 19:10 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1123.eqiad.wmnet with reason: host reimage
* 18:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1123.eqiad.wmnet with OS trixie
* 18:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1109.eqiad.wmnet with OS trixie
* 18:56 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1092.eqiad.wmnet with OS trixie
* 18:52 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 18:49 swfrench@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin ([[phab:T430909|T430909]])
* 18:40 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 18:38 swfrench@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo ([[phab:T430909|T430909]])
* 18:11 swfrench-wmf: restarted eqsin, codfw confds - [[phab:T430909|T430909]]
* 18:01 swfrench-wmf: restarted navtiming on webperf2003 - [[phab:T430909|T430909]]
* 17:59 swfrench-wmf: restarted ulsfo confds, confirmed now connected to eqiad backends - [[phab:T430909|T430909]]
* 17:52 sukhe: restart pybal on lvs2011 to switch from conf2004 to conf1008: [[phab:T430909|T430909]]
* 17:51 sukhe: restart pybal on lvs2012 to switch from conf2004 to conf1008 [puppet re-enabled there]: [[phab:T430909|T430909]]
* 17:46 sukhe: restart pybal on lvs2013 to switch from conf2004 to conf1008: [[phab:T430909|T430909]]
* 17:44 swfrench-wmf: switched codfw, eqsin, ulsfo etcd client SRV records to eqiad - [[phab:T430909|T430909]]
* 17:43 swfrench@dns1004: END - running authdns-update
* 17:40 swfrench@dns1004: START - running authdns-update
* 17:40 sukhe: restart pybal on lvs2014 to switch from conf2004 to conf1008: [[phab:T430909|T430909]]
* 17:21 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet
* 17:15 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet
* 17:14 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet
* 17:06 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet
* 17:06 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service'  # lvs2013, [[phab:T416623|T416623]]
* 17:04 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet
* 17:02 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1111.eqiad.wmnet with OS trixie
* 17:00 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service'  # lvs2014, [[phab:T416623|T416623]]
* 16:58 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet
* 16:58 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS bookworm
* 16:55 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service'  # lvs1019, [[phab:T416623|T416623]]
* 16:53 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-eqiad' 'systemctl restart pybal.service'  # lvs1020, [[phab:T416623|T416623]]
* 16:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1111.eqiad.wmnet with reason: host reimage
* 16:40 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1111.eqiad.wmnet with reason: host reimage
* 16:38 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 47794
* 16:35 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 47794
* 16:27 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1111.eqiad.wmnet with OS trixie
* 16:21 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1006.eqiad.wmnet with OS trixie
* 16:06 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 16:03 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1006.eqiad.wmnet with reason: host reimage
* 15:58 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1121.eqiad.wmnet with OS trixie
* 15:58 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 15:56 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1006.eqiad.wmnet with reason: host reimage
* 15:54 mutante: jenkins down in planned maintenance window
* 15:42 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1037.eqiad.wmnet
* 15:42 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet
* 15:42 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet
* 15:41 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1006.eqiad.wmnet with OS trixie
* 15:34 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1121.eqiad.wmnet with reason: host reimage
* 15:33 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm
* 15:33 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS trixie
* 15:30 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1121.eqiad.wmnet with reason: host reimage
* 15:29 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org
* 15:20 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org
* 15:18 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1121
* 15:18 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1121
* 15:18 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1002.wikimedia.org
* 15:17 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1121
* 15:17 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:17 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 15:16 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
* 15:16 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:16 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:16 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1121 - atsuko@cumin1003&quot;
* 15:16 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1121 - atsuko@cumin1003&quot;
* 15:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1037.eqiad.wmnet with OS trixie
* 15:11 atsuko@cumin1003: START - Cookbook sre.dns.netbox
* 15:09 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
* 15:09 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1121
* 15:09 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host clouddumps1002.wikimedia.org
* 15:09 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
* 15:09 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1121.eqiad.wmnet with OS trixie
* 15:08 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1007.eqiad.wmnet with OS trixie
* 15:08 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host clouddumps1002.wikimedia.org
* 15:08 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
* 15:05 brennen@deploy1003: Finished deploy [phabricator/deployment@7e02037]: deploy phab1004 for [[phab:T431440|T431440]] (duration: 00m 47s)
* 15:04 brennen@deploy1003: Started deploy [phabricator/deployment@7e02037]: deploy phab1004 for [[phab:T431440|T431440]]
* 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@7e02037]: deploy phab2003 for [[phab:T431440|T431440]] (duration: 00m 51s)
* 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@7e02037]: deploy phab2003 for [[phab:T431440|T431440]]
* 15:00 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71] (thin): Regular analytics weekly train THIN [analytics/refinery@7d8dc71f] (duration: 02m 10s)
* 14:58 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71] (thin): Regular analytics weekly train THIN [analytics/refinery@7d8dc71f]
* 14:58 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71]: Regular analytics weekly train [analytics/refinery@7d8dc71f] (duration: 04m 14s)
* 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage
* 14:53 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71]: Regular analytics weekly train [analytics/refinery@7d8dc71f]
* 14:53 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7d8dc71f] (duration: 02m 00s)
* 14:51 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7d8dc71f]
* 14:51 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab2003.codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: maintenance
* 14:51 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage
* 14:50 JavierMonton: Deploying Refinery at {{Gerrit|7d8dc71f}} for change {{Gerrit|1308087}} / [[phab:T431318|T431318]] - update filerevision table sqoop and table
* 14:49 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1007.eqiad.wmnet with reason: host reimage
* 14:42 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1007.eqiad.wmnet with reason: host reimage
* 14:40 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1083.eqiad.wmnet with OS trixie
* 14:37 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:36 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:35 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-master@eqiad
* 14:35 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 14:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1037
* 14:34 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1037
* 14:34 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:cleanMentorList.php --wiki=frwiki  # [[phab:T427386|T427386]]
* 14:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 14:34 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308112{{!}}Revert^2 &quot;[Growth] frwiki: Deploy automated mentor list cleaner&quot; (T427386)]] (duration: 06m 47s)
* 14:34 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
* 14:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:33 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:32 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1037
* 14:31 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:31 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:29 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-master@eqiad
* 14:29 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:29 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:29 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:28 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:27 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:27 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1308112{{!}}Revert^2 &quot;[Growth] frwiki: Deploy automated mentor list cleaner&quot; (T427386)]]
* 14:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
* 14:26 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1007.eqiad.wmnet with OS trixie
* 14:26 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:26 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
* 14:26 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication
* 14:25 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:25 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:25 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:cleanMentorList.php --wiki=frwiki  # [[phab:T427386|T427386]]
* 14:24 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:24 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:24 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1037 - blake@cumin1003&quot;
* 14:24 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1037 - blake@cumin1003&quot;
* 14:20 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1083.eqiad.wmnet with reason: host reimage
* 14:19 blake@cumin1003: START - Cookbook sre.dns.netbox
* 14:19 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-master@codfw
* 14:19 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 14:19 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1037
* 14:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1037.eqiad.wmnet with OS trixie
* 14:18 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet
* 14:18 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
* 14:18 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet
* 14:18 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1037.eqiad.wmnet
* 14:16 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2007.codfw.wmnet with OS trixie
* 14:16 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1083.eqiad.wmnet with reason: host reimage
* 14:15 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1036.eqiad.wmnet
* 14:15 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet
* 14:14 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet
* 14:12 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-master@codfw
* 14:11 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1120.eqiad.wmnet with OS trixie
* 14:05 moritzm: installing distro-info-data updates from trixie/bookworm point releases
* 14:04 fabfur: disable puppet on A:cp-text to selectively apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308040
* 14:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308106{{!}}fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105{{!}}fix(MentorListCleaner): Do not access property before inicialization (T430689)]] (duration: 27m 48s)
* 14:00 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 14:00 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1083.eqiad.wmnet with OS trixie
* 13:58 urbanecm@deploy1003: urbanecm: Continuing with deployment
* 13:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1308106{{!}}fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105{{!}}fix(MentorListCleaner): Do not access property before inicialization (T430689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:58 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet with OS bookworm
* 13:57 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2007.codfw.wmnet with reason: host reimage
* 13:57 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
* 13:53 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1120.eqiad.wmnet with reason: host reimage
* 13:50 moritzm: installing Linux 5.10.259 on Bullseye hosts
* 13:47 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 13:47 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
* 13:46 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2007.codfw.wmnet with reason: host reimage
* 13:46 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
* 13:46 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1120.eqiad.wmnet with reason: host reimage
* 13:46 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:46 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:44 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
* 13:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:38 moritzm: installing e2fsprogs updates from Trixie point release
* 13:35 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1308106{{!}}fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105{{!}}fix(MentorListCleaner): Do not access property before inicialization (T430689)]]
* 13:33 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1120.eqiad.wmnet with OS trixie
* 13:33 topranks: reset cr3-eqsin configuration so traffic uses it again after upgrade
* 13:32 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1088.eqiad.wmnet with OS trixie
* 13:32 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS trixie
* 13:32 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns7002.*
* 13:29 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2007.codfw.wmnet with OS trixie
* 13:23 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1004.eqiad.wmnet with reason: host reimage
* 13:20 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2006.codfw.wmnet with OS trixie
* 13:18 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1036.eqiad.wmnet with OS trixie
* 13:17 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1004.eqiad.wmnet with reason: host reimage
* 13:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 13:16 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 13:15 jayme: Istio is being upgraded from 1.24.2 to 1.29.4 on wikikube staging eqiad and codfw - [[phab:T427401|T427401]]
* 13:14 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1087.eqiad.wmnet with OS trixie
* 13:14 topranks: reboot cr3-eqsin to install new JunOS and set PIC 0/0/0 to 100G
* 13:14 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage
* 13:13 jmm@dns1004: END - running authdns-update
* 13:12 jmm@dns1004: START - running authdns-update
* 13:09 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage
* 13:07 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1082.eqiad.wmnet with OS trixie
* 13:07 jmm@dns1004: END - running authdns-update
* 13:06 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1004.eqiad.wmnet with OS bookworm
* 13:05 jmm@dns1004: START - running authdns-update
* 13:02 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2006.codfw.wmnet with reason: host reimage
* 12:58 topranks: load updated JunOS on cr3-eqsin [[phab:T429386|T429386]]
* 12:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage
* 12:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet
* 12:57 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2006.codfw.wmnet with reason: host reimage
* 12:57 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: upgrade JunOS cr3-eqsin
* 12:56 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs[5004-5006].eqsin.wmnet with reason: upgrade JunOS cr3-eqsin
* 12:55 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 12:55 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 12:53 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage
* 12:53 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 12:52 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1088.eqiad.wmnet with OS trixie
* 12:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet
* 12:52 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:51 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet
* 12:49 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage
* 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 12:48 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage
* 12:44 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1082.eqiad.wmnet with reason: host reimage
* 12:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet
* 12:42 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 12:42 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 12:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply&lt;nowiki&gt;}&lt;/nowiki&gt;
* 12:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply
* 12:39 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:39 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: move dumps-nfs IP to the shared one - filippo@cumin1003&quot;
* 12:39 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: move dumps-nfs IP to the shared one - filippo@cumin1003&quot;
* 12:39 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2006.codfw.wmnet with OS trixie
* 12:38 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1082.eqiad.wmnet with reason: host reimage
* 12:36 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 12:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 12:33 filippo@cumin1003: START - Cookbook sre.dns.netbox
* 12:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1036
* 12:32 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1036
* 12:32 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2005.codfw.wmnet with OS trixie
* 12:32 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1087.eqiad.wmnet with OS trixie
* 12:30 jmm@dns1004: END - running authdns-update
* 12:29 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1036
* 12:29 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1036.eqiad.wmnet 21.32.64.10.in-addr.arpa 1.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:29 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1036.eqiad.wmnet 21.32.64.10.in-addr.arpa 1.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:29 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:29 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1036 - blake@cumin1003&quot;
* 12:29 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1036 - blake@cumin1003&quot;
* 12:28 jmm@dns1004: START - running authdns-update
* 12:26 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:26 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:23 blake@cumin1003: START - Cookbook sre.dns.netbox
* 12:23 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1036
* 12:23 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1036.eqiad.wmnet with OS trixie
* 12:22 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet
* 12:22 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1082.eqiad.wmnet with OS trixie
* 12:22 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet
* 12:22 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1036.eqiad.wmnet
* 12:21 marostegui: Restart mariadb@s7 on db1155 to pick up new filters - [[phab:T431124|T431124]]
* 12:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: restarting for replication filter
* 12:20 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:14 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:14 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2005.codfw.wmnet with reason: host reimage
* 12:14 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 12:08 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:07 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:07 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2005.codfw.wmnet with reason: host reimage
* 12:06 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:06 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:06 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:05 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:05 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:04 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-master-eqiad
* 12:04 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl1002.eqiad.wmnet
* 12:04 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl1002.eqiad.wmnet
* 12:04 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:04 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:03 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:03 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 12:03 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 12:03 brouberol@cumin1003: START - Cookbook sre.wdqs.restart
* 11:59 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl1002.eqiad.wmnet
* 11:59 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl1002.eqiad.wmnet
* 11:59 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:59 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:56 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:56 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl1001.eqiad.wmnet
* 11:56 klausman@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-master-eqiad
* 11:50 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2005.codfw.wmnet with OS trixie
* 11:49 blake@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on wikikube-worker1160.eqiad.wmnet with reason: Verifying matchers for silence
* 11:42 topranks: cr3-eqsin, begin traffic drain to reset PIC and upgrade JunOS [[phab:T429386|T429386]]
* 11:41 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs[5004-5006].eqsin.wmnet with reason: upgrade JunOS cr3-eqsin
* 11:39 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: upgrade JunOS cr3-eqsin
* 11:36 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=thanos-fe2004.codfw.wmnet
* 11:35 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1086.eqiad.wmnet with OS trixie
* 11:35 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=thanos-fe2004.codfw.wmnet
* 11:32 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1085.eqiad.wmnet with OS trixie
* 11:14 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1086.eqiad.wmnet with reason: host reimage
* 11:10 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1085.eqiad.wmnet with reason: host reimage
* 11:04 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2004.codfw.wmnet with OS trixie
* 11:03 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1086.eqiad.wmnet with reason: host reimage
* 11:02 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1085.eqiad.wmnet with reason: host reimage
* 10:48 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2004.codfw.wmnet with reason: host reimage
* 10:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:46 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1086.eqiad.wmnet with OS trixie
* 10:46 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1085.eqiad.wmnet with OS trixie
* 10:44 cgoubert@deploy1003: Finished deploy [restbase/deploy@2fc37d4]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]] (duration: 16m 44s)
* 10:43 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2004.codfw.wmnet with reason: host reimage
* 10:35 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:27 cgoubert@deploy1003: Started deploy [restbase/deploy@2fc37d4]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]]
* 10:27 cgoubert@deploy1003: Finished deploy [restbase/deploy@8a25036]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]] (duration: 00m 45s)
* 10:26 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1117.eqiad.wmnet with OS trixie
* 10:26 cgoubert@deploy1003: Started deploy [restbase/deploy@8a25036]: {{Gerrit|1306049}}: Add isvwiki to RESTBase {{!}} https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/deploy/+/1306049 - [[phab:T429936|T429936]]
* 10:26 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host thanos-fe2004
* 10:25 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host thanos-fe2004
* 10:22 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1116.eqiad.wmnet with OS trixie
* 10:21 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host thanos-fe2004
* 10:21 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) thanos-fe2004.codfw.wmnet 157.32.192.10.in-addr.arpa 7.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:20 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache thanos-fe2004.codfw.wmnet 157.32.192.10.in-addr.arpa 7.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:20 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:20 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host thanos-fe2004 - mvernon@cumin2003&quot;
* 10:20 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host thanos-fe2004 - mvernon@cumin2003&quot;
* 10:15 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2252: Repooling after reboot
* 10:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 10:15 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache
* 10:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2252: Repooling after reboot
* 10:15 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1153.eqiad.wmnet
* 10:14 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1153.eqiad.wmnet
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet
* 10:14 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet
* 10:12 mvernon@cumin2003: START - Cookbook sre.dns.netbox
* 10:12 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host thanos-fe2004
* 10:11 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2004.codfw.wmnet with OS trixie
* 10:07 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1117.eqiad.wmnet with reason: host reimage
* 10:03 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1116.eqiad.wmnet with reason: host reimage
* 09:58 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:58 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1117.eqiad.wmnet with reason: host reimage
* 09:57 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1116.eqiad.wmnet with reason: host reimage
* 09:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 41 days, 15:00:00 on db2252.codfw.wmnet with reason: Security updates
* 09:45 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1117.eqiad.wmnet with OS trixie
* 09:45 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1116.eqiad.wmnet with OS trixie
* 09:28 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1153: Security updates
* 09:28 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 09:28 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 09:28 root@cumin1003: START - Cookbook sre.mysql.depool depool db1153: Security updates
* 09:22 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1016: Security updates
* 09:22 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 09:21 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 09:21 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1016: Security updates
* 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 09:14 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:56 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1016: Security updates
* 08:56 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 08:56 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:56 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1016: Security updates
* 08:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
* 08:45 filippo@dns1006: END - running authdns-update
* 08:43 filippo@dns1006: START - running authdns-update
* 08:42 godog: switch dumps-nfs address to be shared with rsync/http - [[phab:T411248|T411248]]
* 08:40 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1016: Security updates
* 08:40 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 08:40 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:40 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1016: Security updates
* 08:29 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host cirrussearch1111.eqiad.wmnet
* 08:29 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:27 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:27 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:25 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:25 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:24 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:17 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet
* 08:17 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 08:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1015: Security updates
* 08:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 08:09 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 08:09 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1015: Security updates
* 07:42 Msz2001: Deployed private patch for Suggested Ivestigations
* 07:41 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1015: Security updates
* 07:41 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 07:41 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 07:41 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1015: Security updates
* 07:40 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 07:11 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fingerprint warnings - oblivian@cumin1003&quot;
* 07:11 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fingerprint warnings - oblivian@cumin1003
* 07:11 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1024: Security updates
* 07:11 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 07:11 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 07:11 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1024: Security updates
* 07:10 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fingerprint warnings - oblivian@cumin1003
* 07:10 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fingerprint warnings - oblivian@cumin1003&quot;
* 06:55 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host cirrussearch1111.eqiad.wmnet
* 06:55 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet
* 06:48 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1024: Security updates
* 06:48 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 06:48 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 06:48 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1024: Security updates
* 06:42 moritzm: install nginx security updates
* 06:31 root@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool pc1024: Security updates
* 06:21 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1024: Security updates
* 06:19 moritzm: installing php8.2 security updates
* 06:15 moritzm: installing php8.4 security updates
* 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.7 (duration: 02m 38s)
* 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.10  refs [[phab:T430829|T430829]] (duration: 37m 04s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.10  refs [[phab:T430829|T430829]]
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 51s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-06 ==
* 23:30 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306983{{!}}Remove unused user skin preference config (T358273)]], [[gerrit:1306982{{!}}Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921{{!}}Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453{{!}}Drop unused VectorNightMode config (T393977)]] (duration: 09m 39s)
* 23:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1078.eqiad.wmnet with OS trixie
* 23:26 jdlrobson@deploy1003: jdlrobson, bwang: Continuing with deployment
* 23:22 jdlrobson@deploy1003: jdlrobson, bwang: Backport for [[gerrit:1306983{{!}}Remove unused user skin preference config (T358273)]], [[gerrit:1306982{{!}}Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921{{!}}Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453{{!}}Drop unused VectorNightMode config (T393977)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:21 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1306983{{!}}Remove unused user skin preference config (T358273)]], [[gerrit:1306982{{!}}Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921{{!}}Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453{{!}}Drop unused VectorNightMode config (T393977)]]
* 23:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1078.eqiad.wmnet with reason: host reimage
* 23:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1078.eqiad.wmnet with reason: host reimage
* 22:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1078.eqiad.wmnet with OS trixie
* 22:29 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cirrussearch1114.eqiad.wmnet with reason: reimage on hold until restore completes
* 22:22 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cirrussearch[1079,1115].eqiad.wmnet with reason: reimage on hold until restore completes
* 21:18 maryum: Deployed security fix for [[phab:T428006|T428006]]
* 20:44 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1079.eqiad.wmnet with OS trixie
* 20:33 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1077.eqiad.wmnet with OS trixie
* 20:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1115.eqiad.wmnet with OS trixie
* 20:25 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1079.eqiad.wmnet with reason: host reimage
* 20:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1079.eqiad.wmnet with reason: host reimage
* 20:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1303490{{!}}T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812{{!}}Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] (duration: 08m 14s)
* 20:14 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1077.eqiad.wmnet with reason: host reimage
* 20:10 krinkle@deploy1003: krinkle, pushpaktiwari: Continuing with deployment
* 20:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1115.eqiad.wmnet with reason: host reimage
* 20:08 krinkle@deploy1003: krinkle, pushpaktiwari: Backport for [[gerrit:1303490{{!}}T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812{{!}}Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1077.eqiad.wmnet with reason: host reimage
* 20:06 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1303490{{!}}T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812{{!}}Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]]
* 20:05 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1079.eqiad.wmnet with OS trixie
* 20:04 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1115.eqiad.wmnet with reason: host reimage
* 19:52 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1077.eqiad.wmnet with OS trixie
* 19:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1115.eqiad.wmnet with OS trixie
* 19:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 19:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 18:59 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1114.eqiad.wmnet with OS trixie
* 18:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1114.eqiad.wmnet with reason: host reimage
* 18:35 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1114.eqiad.wmnet with reason: host reimage
* 18:32 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1112.eqiad.wmnet with OS trixie
* 18:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1114.eqiad.wmnet with OS trixie
* 18:21 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1072.eqiad.wmnet with OS trixie
* 18:14 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1112.eqiad.wmnet with reason: host reimage
* 18:10 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1112.eqiad.wmnet with reason: host reimage
* 17:59 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1072.eqiad.wmnet with reason: host reimage
* 17:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1112.eqiad.wmnet with OS trixie
* 17:55 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1072.eqiad.wmnet with reason: host reimage
* 17:39 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1072.eqiad.wmnet with OS trixie
* 17:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1071.eqiad.wmnet with OS trixie
* 17:18 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1070.eqiad.wmnet with OS trixie
* 17:16 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1084.eqiad.wmnet with OS trixie
* 16:54 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1071.eqiad.wmnet with reason: host reimage
* 16:53 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1084.eqiad.wmnet with reason: host reimage
* 16:51 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1070.eqiad.wmnet with reason: host reimage
* 16:49 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1084.eqiad.wmnet with reason: host reimage
* 16:38 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1071.eqiad.wmnet with OS trixie
* 16:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1096.eqiad.wmnet with OS trixie
* 16:35 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1070.eqiad.wmnet with OS trixie
* 16:33 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1084.eqiad.wmnet with OS trixie
* 16:30 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1089.eqiad.wmnet with OS trixie
* 16:28 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1103.eqiad.wmnet with OS trixie
* 16:19 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1096.eqiad.wmnet with reason: host reimage
* 16:14 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1096.eqiad.wmnet with reason: host reimage
* 16:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1089.eqiad.wmnet with reason: host reimage
* 16:05 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage
* 16:02 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1003.eqiad.wmnet with OS bookworm
* 16:01 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1089.eqiad.wmnet with reason: host reimage
* 16:00 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage
* 15:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1096.eqiad.wmnet with OS trixie
* 15:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1080.eqiad.wmnet with OS trixie
* 15:46 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1089.eqiad.wmnet with OS trixie
* 15:45 dancy@deploy1003: Installation of scap version &quot;4.272.0&quot; completed for 158 hosts
* 15:43 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1103.eqiad.wmnet with OS trixie
* 15:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1113.eqiad.wmnet with OS trixie
* 15:41 dancy@deploy1003: Installing scap version &quot;4.272.0&quot; for 158 host(s)
* 15:40 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 15:39 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 15:38 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 15:38 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1069.eqiad.wmnet with OS trixie
* 15:37 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 15:36 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:34 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 15:33 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage
* 15:27 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage
* 15:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1113.eqiad.wmnet with reason: host reimage
* 15:19 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1069.eqiad.wmnet with reason: host reimage
* 15:18 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1113.eqiad.wmnet with reason: host reimage
* 15:16 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1069.eqiad.wmnet with reason: host reimage
* 15:11 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:11 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1080.eqiad.wmnet with OS trixie
* 15:11 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 15:05 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1113.eqiad.wmnet with OS trixie
* 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1003.eqiad.wmnet with reason: host reimage
* 14:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1003.eqiad.wmnet with OS bookworm
* 14:33 elukey: rolled out spicerack on all cumin nodes - [[phab:T429699|T429699]]
* 14:32 elukey: upgrade all bookworm hosts to pywmflib 3.1 - [[phab:T430552|T430552]]
* 14:14 marostegui: Setup x4 eqiad topology [[phab:T404715|T404715]]
* 14:13 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 14:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db2230.codfw.wmnet
* 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db2230.codfw.wmnet
* 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[2001-2002].codfw.wmnet
* 13:51 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 13:45 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97)
* 13:45 cwilliams@cumin1003: dbmaint on s4@codfw [[phab:T429893|T429893]]
* 13:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 13:42 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-master-codfw
* 13:42 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl2002.codfw.wmnet
* 13:42 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl2002.codfw.wmnet
* 13:38 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl2002.codfw.wmnet
* 13:38 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl2002.codfw.wmnet
* 13:38 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl2001.codfw.wmnet
* 13:38 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl2001.codfw.wmnet
* 13:35 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl2001.codfw.wmnet
* 13:35 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl2001.codfw.wmnet
* 13:35 klausman@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-master-codfw
* 12:30 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1265672{{!}}robots.php: Change Beta Cluster override from prepend to replace]] (duration: 25m 11s)
* 12:24 krinkle@deploy1003: krinkle: Continuing with deployment
* 12:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet
* 12:09 krinkle@deploy1003: krinkle: Backport for [[gerrit:1265672{{!}}robots.php: Change Beta Cluster override from prepend to replace]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:08 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2048.codfw.wmnet
* 12:05 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1265672{{!}}robots.php: Change Beta Cluster override from prepend to replace]]
* 11:57 moritzm: installing curl security updates
* 11:49 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 11:31 moritzm: installing nano security updates
* 11:07 moritzm: failover Ganeti master in codfw to ganeti2032 [[phab:T430909|T430909]]
* 11:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 11:04 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1005.eqiad.wmnet with OS trixie
* 11:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:50 jmm@dns1004: END - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:46 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:44 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1005.eqiad.wmnet with reason: host reimage
* 10:38 elukey@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1005.eqiad.wmnet with reason: host reimage
* 10:31 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:31 marostegui: Setup x4 codfw topology [[phab:T404715|T404715]]
* 10:31 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:24 elukey: spicerack 13.0.0 deployed on cumin2002
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:23 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:23 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 10:21 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1005.eqiad.wmnet with OS trixie
* 10:20 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 10:17 elukey: uploaded spicerack_13.0.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia
* 09:54 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:52 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:20 elukey: upgrade all bullseye hosts to pywmflib 3.1 - [[phab:T430552|T430552]]
* 09:10 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s4
* 09:10 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s6
* 09:07 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 08:58 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 08:56 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 08:56 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 08:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2002.codfw.wmnet
* 08:06 godog: remove cloudvirt1046, cloudvirt1062, cloudvirt1074, cloudvirt1075 from maintenance aggregate and put them in network-ovs - [[phab:T424802|T424802]]
* 08:00 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet
* 07:58 hashar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307617{{!}}Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] (duration: 32m 53s)
* 07:57 fabfur: repooled cp4038
* 07:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp4038.*
* 07:53 moritzm: installing pyjwt security updates
* 07:47 moritzm: installing openjpeg2 security updates
* 07:45 hashar@deploy1003: vadymts1, hashar: Continuing with deployment
* 07:43 hashar@deploy1003: vadymts1, hashar: Backport for [[gerrit:1307617{{!}}Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:38 moritzm: installing python-urllib3 security updates
* 07:37 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
* 07:30 fabfur: depooled cp4038 to investigate on possible maxmind failure
* 07:30 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp4038.*
* 07:30 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp4038.*
* 07:29 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw
* 07:25 hashar@deploy1003: Started scap sync-world: Backport for [[gerrit:1307617{{!}}Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]]
* 06:13 moritzm: installing Linux 6.12.95 on trixie hosts
* 05:20 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s6
* 05:20 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s4
* 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1015.eqiad.wmnet with reason: cloning
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 08s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-05 ==
* 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 08s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-04 ==
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 58s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-03 ==
* 17:08 topranks: revert protocol preference changes on cr3-ulsfo after upgrade
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr2-eqord with reason: upgrade JunOS cr3-ulsfo
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr4-ulsfo with reason: upgrade JunOS cr3-ulsfo
* 16:48 topranks: reboot cr3-ulsfo to upgrade JunOS and reset linecard [[phab:T424839|T424839]]
* 15:52 topranks: adjust outbound BGP policies on cr3-ulsfo to drain router of traffic [[phab:T424839|T424839]]
* 15:45 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs[4008-4010].ulsfo.wmnet with reason: upgrade JunOS cr3-ulsfo
* 15:44 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw1-[22-23]-ulsfo,cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: upgrade JunOS cr3-ulsfo
* 15:36 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 14:40 cmooney@dns3003: END - running authdns-update
* 14:26 cmooney@dns3003: START - running authdns-update
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:19 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:16 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 13:38 sukhe@dns1004: END - running authdns-update
* 13:35 sukhe@dns1004: START - running authdns-update
* 13:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:26 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:18 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:17 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:14 moritzm: imported samplicator 1.3.8rc1-1+deb13u1 to trixie-wikimedia/main [[phab:T337208|T337208]]
* 13:13 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:07 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:07 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:02 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:02 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:00 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:58 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:53 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:52 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:50 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 12:50 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 12:47 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:41 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:39 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:32 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 12:26 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 12:23 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 12:19 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[2004-2007].codfw.wmnet
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:15 jynus@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:09 jynus@cumin2003: START - Cookbook sre.dns.netbox
* 11:58 jynus@cumin2003: START - Cookbook sre.hosts.decommission for hosts backup[2004-2007].codfw.wmnet
* 10:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[1004-1007].eqiad.wmnet
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 10:01 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 09:52 jynus@cumin1003: START - Cookbook sre.dns.netbox
* 09:39 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:36 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts backup[1004-1007].eqiad.wmnet
* 09:36 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 09:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 09:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:57 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:55 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 08:49 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 08:49 atsukoito: depooling cirrussearch in codfw because of regression after upgrade [[phab:T431091|T431091]]
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mirror1001.wikimedia.org
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:29 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:18 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 08:11 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts mirror1001.wikimedia.org
* 06:15 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 18s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-02 ==
* 22:55 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1003.wikimedia.org with OS trixie
* 22:29 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:23 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:05 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1003.wikimedia.org with OS trixie
* 22:03 mutante: contint1003 (zuul.wikimedia.org) - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 22:03 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on zuul.wikimedia.org with reason: reimage
* 21:39 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 18s)
* 21:39 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 21:20 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 21:19 sbassett: Deployed security fix for [[phab:T428829|T428829]]
* 20:58 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:55 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:40 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] (duration: 12m 35s)
* 20:36 arlolra@deploy1003: cscott, arlolra: Continuing with deployment
* 20:35 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 20s)
* 20:35 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 20:33 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint2003.wikimedia.org with OS trixie
* 20:31 arlolra@deploy1003: cscott, arlolra: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Cha
* 20:28 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]]
* 20:17 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] (duration: 08m 13s)
* 20:14 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:13 sbassett@deploy1003: sbassett: Continuing with deployment
* 20:11 sbassett@deploy1003: sbassett: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:09 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]]
* 20:08 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 20:08 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 20:08 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:05 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 19:49 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint2003.wikimedia.org with OS trixie
* 19:48 mutante: contint2003 - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 18:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:13 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 17:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1003.eqiad.wmnet with OS bookworm
* 17:52 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:52 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:51 jasmine@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=wikikube-ctrl1005.eqiad.wmnet
* 17:48 jasmine_: homer &quot;cr*eqiad*&quot; commit &quot;Added new stacked control plane wikikube-ctrl1005&quot;
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
* 17:31 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] (duration: 09m 33s)
* 17:26 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 17:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:23 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:21 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]]
* 17:18 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:16 rscout@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:15 rscout@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:12 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wcqs[2002-2003].codfw.wmnet,wcqs1002.eqiad.wmnet with reason: reimaging hosts
* 17:08 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 17:07 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 17:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:03 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:03 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1003
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1003
* 17:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1003.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:57 tappof: bump space for prometheus k8s-aux in eqiad
* 16:55 cmooney@dns3003: END - running authdns-update
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:53 cmooney@dns3003: START - running authdns-update
* 16:52 ryankemper: [ml-serve-eqiad] Cleared out 1302 failed (Evicted) pods: `kubectl -n llm delete pods --field-selector=status.phase=Failed`, freeing calico-kube-controllers from OOM crashloop (evictions were caused by disk pressure)
* 16:49 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:46 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 16:39 rzl@dns1004: END - running authdns-update
* 16:37 rzl@dns1004: START - running authdns-update
* 16:36 rzl@dns1004: START - running authdns-update
* 16:35 rzl@deploy1003: Finished scap sync-world: [[phab:T416623|T416623]] (duration: 10m 19s)
* 16:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:33 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:30 rzl@deploy1003: rzl: Continuing with deployment
* 16:28 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:26 rzl@deploy1003: rzl: [[phab:T416623|T416623]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:25 rzl@deploy1003: Started scap sync-world: [[phab:T416623|T416623]]
* 16:25 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: sync
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: sync
* 16:16 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1003.eqiad.wmnet with OS bookworm
* 16:11 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:11 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 16:08 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 16:08 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Security updates
* 15:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:45 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:45 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Security updates
* 15:42 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1003.eqiad.wmnet with OS bookworm
* 15:24 moritzm: installing busybox updates from bookworm point release
* 15:20 moritzm: installing busybox updates from trixie point release
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:15 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:15 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Security updates
* 15:13 moritzm: installing giflib security updates
* 15:08 moritzm: installing Tomcat security updates
* 14:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 14:56 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 14:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:53 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 14:53 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 14:53 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Security updates
* 14:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:52 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94711 and previous config saved to /var/cache/conftool/dbconfig/20260702-144644-fceratto.json
* 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94709 and previous config saved to /var/cache/conftool/dbconfig/20260702-143636-fceratto.json
* 14:32 moritzm: installing libdbi-perl security updates
* 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94708 and previous config saved to /var/cache/conftool/dbconfig/20260702-142628-fceratto.json
* 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94707 and previous config saved to /var/cache/conftool/dbconfig/20260702-141621-fceratto.json
* 14:12 moritzm: installing rsync security updates
* 14:11 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94706 and previous config saved to /var/cache/conftool/dbconfig/20260702-140959-fceratto.json
* 14:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2205.codfw.wmnet with reason: Maintenance
* 14:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 14:07 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 Tran: Deployed patch for [[phab:T427287|T427287]]
* 14:04 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:59 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 13:59 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2205 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94704 and previous config saved to /var/cache/conftool/dbconfig/20260702-135505-fceratto.json
* 13:54 moritzm: installing sed security updates
* 13:53 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:52 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2209 to s3 primary [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94703 and previous config saved to /var/cache/conftool/dbconfig/20260702-135235-fceratto.json
* 13:52 federico3: Starting s3 codfw failover from db2205 to db2209 - [[phab:T430912|T430912]]
* 13:51 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:51 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2209 with weight 0 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94702 and previous config saved to /var/cache/conftool/dbconfig/20260702-134719-fceratto.json
* 13:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [[phab:T430912|T430912]]
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:40 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:37 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 13:36 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:36 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:34 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 13:30 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:26 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:25 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
* 13:23 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:22 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 13:17 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:17 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org
* 13:12 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart (exit_code=97) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=97) rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:09 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] (duration: 07m 20s)
* 13:05 aude@deploy1003: jdrewniak, aude: Continuing with deployment
* 13:04 aude@deploy1003: jdrewniak, aude: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:02 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]]
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs-categories1001.eqiad.wmnet
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 12:10 jmm@dns1004: END - running authdns-update
* 12:07 jmm@dns1004: START - running authdns-update
* 11:51 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 11:44 btullis@cumin1003: START - Cookbook sre.dns.netbox
* 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:42 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 11:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 11:39 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts wdqs-categories1001.eqiad.wmnet
* 11:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 11:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 11:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 11:29 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:29 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 10:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Repooling
* 10:49 jmm@dns1004: END - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94698 and previous config saved to /var/cache/conftool/dbconfig/20260702-103146-fceratto.json
* 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94696 and previous config saved to /var/cache/conftool/dbconfig/20260702-102137-fceratto.json
* 10:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1017.eqiad.wmnet
* 10:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 10:18 fceratto@cumin1003: Removing es1033 from zarcillo [[phab:T408772|T408772]]
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1033.eqiad.wmnet
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:14 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:13 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1017.eqiad.wmnet
* 10:12 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling
* 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94693 and previous config saved to /var/cache/conftool/dbconfig/20260702-101130-fceratto.json
* 10:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:10 fceratto@cumin1003: START - Cookbook sre.dns.netbox
* 10:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:03 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts es1033.eqiad.wmnet
* 10:03 fceratto@cumin1003: START - Cookbook sre.mysql.decommission
* 10:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94691 and previous config saved to /var/cache/conftool/dbconfig/20260702-100122-fceratto.json
* 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94690 and previous config saved to /var/cache/conftool/dbconfig/20260702-095529-fceratto.json
* 09:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 09:54 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:53 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover
* 09:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:44 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2213 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94688 and previous config saved to /var/cache/conftool/dbconfig/20260702-093859-fceratto.json
* 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94687 and previous config saved to /var/cache/conftool/dbconfig/20260702-093650-fceratto.json
* 09:36 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T430923|T430923]]
* 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94686 and previous config saved to /var/cache/conftool/dbconfig/20260702-093004-fceratto.json
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2192 with weight 0 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94685 and previous config saved to /var/cache/conftool/dbconfig/20260702-092455-fceratto.json
* 09:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 [[phab:T430923|T430923]]
* 09:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94684 and previous config saved to /var/cache/conftool/dbconfig/20260702-091957-fceratto.json
* 09:16 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] (duration: 06m 57s)
* 09:13 moritzm: installing libgcrypt20 security updates
* 09:12 kharlan@deploy1003: kharlan: Continuing with deployment
* 09:11 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94683 and previous config saved to /var/cache/conftool/dbconfig/20260702-090950-fceratto.json
* 09:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]]
* 09:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:01 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] (duration: 07m 07s)
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94682 and previous config saved to /var/cache/conftool/dbconfig/20260702-085942-fceratto.json
* 08:57 kharlan@deploy1003: kharlan: Continuing with deployment
* 08:56 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]]
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94681 and previous config saved to /var/cache/conftool/dbconfig/20260702-085237-fceratto.json
* 08:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 08:43 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:40 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:25 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] (duration: 11m 44s)
* 08:21 cscott@deploy1003: cscott: Continuing with deployment
* 08:16 cscott@deploy1003: cscott: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:14 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]]
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Migration of db1244.eqiad.wmnet completed
* 08:02 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:02 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:01 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] (duration: 18m 58s)
* 08:01 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:01 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 07:59 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:59 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2006.wikimedia.org
* 07:58 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:57 cscott@deploy1003: cscott: Continuing with deployment
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:54 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2006.wikimedia.org
* 07:54 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:54 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:44 cscott@deploy1003: cscott: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 07:44 moritzm: installing node-lodash security updates
* 07:42 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]]
* 07:39 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 07:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] (duration: 07m 28s)
* 07:26 cscott@deploy1003: ssastry, cscott: Continuing with deployment
* 07:25 cscott@deploy1003: ssastry, cscott: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:23 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1244: Migration of db1244.eqiad.wmnet completed
* 07:22 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]]
* 07:16 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] (duration: 06m 55s)
* 07:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1244.eqiad.wmnet with OS trixie
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:09 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]]
* 06:54 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:50 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1250.eqiad.wmnet with OS trixie
* 06:34 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1244.eqiad.wmnet with OS trixie
* 06:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: dbmaint on s4@eqiad [[phab:T429893|T429893]]
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:14 cwilliams@dns1006: END - running authdns-update
* 06:12 cwilliams@dns1006: START - running authdns-update
* 06:11 cwilliams@dns1006: END - running authdns-update
* 06:11 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1244 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94676 and previous config saved to /var/cache/conftool/dbconfig/20260702-061059-cwilliams.json
* 06:09 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:09 cwilliams@dns1006: START - running authdns-update
* 06:08 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0)  on VRTS host vrts1003.eqiad.wmnet
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1160 to s4 primary and set section read-write [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94675 and previous config saved to /var/cache/conftool/dbconfig/20260702-060746-cwilliams.json
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94674 and previous config saved to /var/cache/conftool/dbconfig/20260702-060704-cwilliams.json
* 06:06 cezmunsta: Starting s4 eqiad failover from db1244 to db1160 - [[phab:T430817|T430817]]
* 06:04 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade  on VRTS host vrts1003.eqiad.wmnet
* 05:59 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1160 with weight 0 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94673 and previous config saved to /var/cache/conftool/dbconfig/20260702-055927-cwilliams.json
* 05:59 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430817|T430817]]
* 05:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1250.eqiad.wmnet with OS trixie
* 05:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1250.eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 05:39 marostegui: Failover m3 (phabricator) from db1250 to db1228 - [[phab:T430158|T430158]]
* 05:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 04:45 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] (duration: 09m 08s)
* 04:41 tstarling@deploy1003: tstarling, reedy: Continuing with deployment
* 04:38 tstarling@deploy1003: tstarling, reedy: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:36 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]]
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 59s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:16 ryankemper: [[phab:T429844|T429844]] [opensearch] completed `cirrussearch2111` reimage; all codfw search clusters are green, all nodes now report `OpenSearch 2.19.5`, and the temporary chi voting exclusion has been removed
* 00:57 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:29 ryankemper: [[phab:T429844|T429844]] [opensearch] depooled codfw search-omega/search-psi discovery records to match existing codfw search depool during OpenSearch 2.19 migration
* 00:29 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 00:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:01 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:00 ryankemper: [[phab:T429844|T429844]] [opensearch] chi cluster recovered after stopping `opensearch_1@production-search-codfw` on `cirrussearch2111`

== 2026-07-01 ==
* 23:59 ryankemper: [[phab:T429844|T429844]] [opensearch] stopped `opensearch_1@production-search-codfw` on `cirrussearch2111` after chi cluster-manager election churn following `voting_config_exclusions` POST; hoping this triggers a re-election
* 23:52 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 23:50 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2003.codfw.wmnet with OS bookworm
* 22:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 22:13 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2084.codfw.wmnet with OS trixie
* 22:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 22:03 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:01 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 21:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 21:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:36 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:35 bking@cumin2003: START - Cookbook sre.dns.netbox
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2003
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2003.codfw.wmnet with OS bookworm
* 21:19 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2084.codfw.wmnet with OS trixie
* 21:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2108.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:45 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:28 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 20:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 19:59 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2108.codfw.wmnet with OS trixie
* 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2093.codfw.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:43 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2080.codfw.wmnet with OS trixie
* 19:28 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:18 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:17 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 19:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 18:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2093.codfw.wmnet with OS trixie
* 18:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2080.codfw.wmnet with OS trixie
* 18:27 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 18:18 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] (duration: 09m 15s)
* 18:13 jgiannelos@deploy1003: jgiannelos, neriah: Continuing with deployment
* 18:11 jgiannelos@deploy1003: jgiannelos, neriah: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:09 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]]
* 17:40 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 16:58 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 30 hosts
* 16:57 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 30 hosts
* 16:52 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2202.codfw.wmnet
* 16:52 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2202.codfw.wmnet
* 16:51 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2012.codfw.wmnet
* 16:51 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2012.codfw.wmnet
* 16:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2076.codfw.wmnet with OS trixie
* 16:49 brett: Start pybal on lvs2012 - [[phab:T429861|T429861]]
* 16:49 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.remove-downtime (exit_code=97) for 59 hosts
* 16:48 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 59 hosts
* 16:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2061.codfw.wmnet with OS trixie
* 16:30 dancy@deploy1003: Installation of scap version &quot;4.271.0&quot; completed for 2 hosts
* 16:28 dancy@deploy1003: Installing scap version &quot;4.271.0&quot; for 2 host(s)
* 16:23 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:18 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:18 jasmine@dns1004: END - running authdns-update
* 16:16 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:15 jasmine@dns1004: START - running authdns-update
* 16:14 jasmine@dns1004: END - running authdns-update
* 16:12 jasmine@dns1004: START - running authdns-update
* 16:07 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: maintenance
* 16:06 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt with reason: Junos upograde
* 16:00 papaul: ongoing maintenance on lsw1-b2-codfw
* 16:00 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2076.codfw.wmnet with OS trixie
* 15:59 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:59 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:57 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2061.codfw.wmnet with OS trixie
* 15:55 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:55 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:51 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:51 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Repooling after switchover
* 15:50 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:50 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 15:48 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2092.codfw.wmnet with OS trixie
* 15:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:30 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 15:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:22 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: Rack B2 maintenance - [[phab:T429861|T429861]]
* 15:21 brett: Stopping pybal on lvs2012 in preparation for codfw rack b2 maintenance - [[phab:T429861|T429861]]
* 15:20 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:12 _joe_: restarted manually alertmanager-irc-relay
* 15:12 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt with reason: Junos upograde
* 15:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 15:07 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:06 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:02 papaul: ongoing maintenance on lsw1-a8-codfw
* 14:31 topranks: POWERING DOWN CR1-EQIAD for line card installation [[phab:T426343|T426343]]
* 14:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] (duration: 08m 57s)
* 14:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
* 14:24 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:22 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]]
* 14:22 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:16 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:15 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:14 topranks: re-enable routing-engine graceful-failover on cr1-eqiad [[phab:T417873|T417873]]
* 14:13 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:13 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2220: Repooling after switchover
* 14:12 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:11 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:08 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] (duration: 10m 01s)
* 14:07 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2220 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94664 and previous config saved to /var/cache/conftool/dbconfig/20260701-140729-fceratto.json
* 14:06 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2159 to s7 primary [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94663 and previous config saved to /var/cache/conftool/dbconfig/20260701-140503-fceratto.json
* 14:04 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:04 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 14:04 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 14:04 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:04 dreamyjazz@deploy1003: anzx, dreamyjazz: Continuing with deployment
* 14:04 federico3: Starting s7 codfw failover from db2220 to db2159 - [[phab:T430826|T430826]]
* 14:03 jmm@dns1004: END - running authdns-update
* 14:03 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:03 topranks: flipping cr1-eqiad active routing-enginer back to RE0 [[phab:T417873|T417873]]
* 14:03 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudsw1-c8-eqiad,cloudsw1-d5-eqiad with reason: router upgrades eqiad
* 14:01 jmm@dns1004: START - running authdns-update
* 14:00 dreamyjazz@deploy1003: anzx, dreamyjazz: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2159 with weight 0 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94662 and previous config saved to /var/cache/conftool/dbconfig/20260701-135906-fceratto.json
* 13:58 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]]
* 13:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 [[phab:T430826|T430826]]
* 13:56 topranks: reboot routing-enginer RE0 on cr1-eqiad [[phab:T417873|T417873]]
* 13:48 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1006.wikimedia.org
* 13:44 atsuko@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:43 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1006.wikimedia.org
* 13:41 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1005.wikimedia.org
* 13:37 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1005.wikimedia.org
* 13:37 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on pfw1-eqiad with reason: router upgrades eqiad
* 13:35 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[1017-1020].eqiad.wmnet with reason: router upgrades eqiad
* 13:34 caro@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] (duration: 07m 59s)
* 13:30 caro@deploy1003: caro: Continuing with deployment
* 13:28 caro@deploy1003: caro: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:27 topranks: route-engine failover cr1-eqiad
* 13:26 caro@deploy1003: Started scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]]
* 13:15 topranks: rebooting routing-engine 1 on cr1-eqiad [[phab:T417873|T417873]]
* 13:13 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] (duration: 08m 29s)
* 13:13 moritzm: installing qemu security updates
* 13:11 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 13:11 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 13:09 jgiannelos@deploy1003: jgiannelos: Continuing with deployment
* 13:08 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:07 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:06 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance
* 13:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2214: Repooling after switchover
* 13:05 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]]
* 13:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling after switchover
* 13:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2214 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94660 and previous config saved to /var/cache/conftool/dbconfig/20260701-130413-fceratto.json
* 13:01 moritzm: installing python3.13 security updates
* 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2229 to s6 primary [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94659 and previous config saved to /var/cache/conftool/dbconfig/20260701-125959-fceratto.json
* 12:59 federico3: Starting s6 codfw failover from db2214 to db2229 - [[phab:T430814|T430814]]
* 12:57 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 13 hosts with reason: router upgrade and line card install
* 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2229 with weight 0 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94658 and previous config saved to /var/cache/conftool/dbconfig/20260701-125149-fceratto.json
* 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Primary switchover s6 [[phab:T430814|T430814]]
* 12:50 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet
* 12:50 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet
* 12:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2100.codfw.wmnet with OS trixie
* 12:38 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2083.codfw.wmnet with OS trixie
* 12:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Migration of db2240.codfw.wmnet completed
* 12:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:00 topranks: drain traffic on cr1-eqiad to allow for line card install and JunOS upgrade [[phab:T426343|T426343]]
* 11:52 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2083.codfw.wmnet with OS trixie
* 11:50 cmooney@dns2005: END - running authdns-update
* 11:49 cmooney@dns2005: START - running authdns-update
* 11:48 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2100.codfw.wmnet with OS trixie
* 11:40 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 11:40 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 11:31 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2240: Migration of db2240.codfw.wmnet completed
* 11:30 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:28 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
* 11:27 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:23 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2240.codfw.wmnet with OS trixie
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:17 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:16 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:16 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:15 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2086.codfw.wmnet with OS trixie
* 11:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2106.codfw.wmnet with OS trixie
* 11:14 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:13 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:12 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:09 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2115.codfw.wmnet with OS trixie
* 11:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 11:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 10:53 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:44 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:44 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2240.codfw.wmnet with OS trixie
* 10:44 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:42 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: dbmaint on s4@codfw [[phab:T429893|T429893]]
* 10:40 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 10:39 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:27 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2240 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94653 and previous config saved to /var/cache/conftool/dbconfig/20260701-102658-cwilliams.json
* 10:26 moritzm: installing nginx security updates
* 10:26 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2086.codfw.wmnet with OS trixie
* 10:23 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2179 to s4 primary [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94652 and previous config saved to /var/cache/conftool/dbconfig/20260701-102356-cwilliams.json
* 10:23 cezmunsta: Starting s4 codfw failover from db2240 to db2179 - [[phab:T430127|T430127]]
* 10:23 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2106.codfw.wmnet with OS trixie
* 10:20 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2115.codfw.wmnet with OS trixie
* 10:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2179 with weight 0 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94651 and previous config saved to /var/cache/conftool/dbconfig/20260701-101531-cwilliams.json
* 10:15 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430127|T430127]]
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 09:39 mszwarc@deploy1003: Synchronized private/SuggestedInvestigationsSignals/SuggestedInvestigationsSignal4n.php: Update SI signal 4n (duration: 06m 08s)
* 09:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:21 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:54 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:38 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:38 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:36 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:21 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] (duration: 36m 11s)
* 08:15 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:09 mszwarc@deploy1003: mszwarc, abi: Continuing with deployment
* 08:03 mszwarc@deploy1003: mszwarc, abi: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:55 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 07:51 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 07:45 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]]
* 07:30 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050] (duration: 00m 22s)
* 07:29 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050]
* 07:28 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050] (duration: 01m 59s)
* 07:28 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] (duration: 07m 19s)
* 07:26 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050]
* 07:26 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050] (duration: 04m 32s)
* 07:24 mszwarc@deploy1003: wmde-fisch, mszwarc: Continuing with deployment
* 07:23 mszwarc@deploy1003: wmde-fisch, mszwarc: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:21 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050]
* 07:21 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050] (duration: 02m 01s)
* 07:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]]
* 07:19 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050]
* 07:13 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] (duration: 09m 13s)
* 07:09 mszwarc@deploy1003: mszwarc, chlod, revi: Continuing with deployment
* 07:06 mszwarc@deploy1003: mszwarc, chlod, revi: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:04 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]]
* 06:55 elukey: upgrade all trixie hosts to pywmflib 3.0 - [[phab:T430552|T430552]]
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:31 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Niharika29 out of all services on: 2453 hosts
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.hiddenparma (exit_code=99) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:01 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1039.eqiad.wmnet with reason: issues
* 05:41 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1027.eqiad.wmnet
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2068.codfw.wmnet with OS trixie
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s2
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s7
* 05:36 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:16 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:15 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:09 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2067.codfw.wmnet with OS trixie
* 04:56 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2068.codfw.wmnet with OS trixie
* 04:49 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:45 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:27 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2067.codfw.wmnet with OS trixie
* 03:47 slyngshede@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Hardware crash
* 03:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:59 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:55 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:35 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:31 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:30 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:26 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:09 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 54s)
* 02:03 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:07 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad back to read-write - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94649 and previous config saved to /var/cache/conftool/dbconfig/20260701-010716-ladsgroup.json
* 01:05 ladsgroup@dns1004: END - running authdns-update
* 01:05 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool es1039 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94648 and previous config saved to /var/cache/conftool/dbconfig/20260701-010551-ladsgroup.json
* 01:03 ladsgroup@dns1004: START - running authdns-update
* 01:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Promote es1035 to es7 primary [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94647 and previous config saved to /var/cache/conftool/dbconfig/20260701-010002-ladsgroup.json
* 00:58 Amir1: Starting es7 eqiad failover from es1039 to es1035 - [[phab:T430765|T430765]]
* 00:53 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es1035 with weight 0 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94646 and previous config saved to /var/cache/conftool/dbconfig/20260701-005329-ladsgroup.json
* 00:53 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Primary switchover es7 [[phab:T430765|T430765]]
* 00:42 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad as read-only for maintenance - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94645 and previous config saved to /var/cache/conftool/dbconfig/20260701-004221-ladsgroup.json
* 00:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2102.codfw.wmnet with OS trixie
* 00:15 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2103.codfw.wmnet with OS trixie
* 00:05 dr0ptp4kt: DEPLOYED Refinery at {{Gerrit|4e7a2b32}} for changes: pageview allowlist {{Gerrit|1305158}} (+min.wikiquote) {{Gerrit|1305162}} (+bol.wikipedia), {{Gerrit|1305156}} (+isv.wikipedia); {{Gerrit|1305980}} (pv allowlist -api.wikimedia, sqoop +isvwiki); sqoop {{Gerrit|1295064}} (+globalimagelinks) {{Gerrit|1295069}} (+filerevision) using scap, then deployed onto HDFS (manual copyToLocal required additionally)

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>1al9ufrjcm816zmqqcgb4ceozwu7adw</sha1>
    </revision>
  </page>
  <page>
    <title>Data Platform/Systems/Refine/Deploy Refinery</title>
    <ns>0</ns>
    <id>20225</id>
    <revision>
      <id>2435510</id>
      <parentid>2433842</parentid>
      <timestamp>2026-07-12T01:04:22Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>lang=bash</comment>
      <origin>2435510</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6761" sha1="p8pcok5b3e3fhlapmh7pwympba9puj3" xml:space="preserve">Refinery is the software infrastructure that is used on the [[Analytics/Cluster|Analytics Cluster]]. The source code is in the [https://gerrit.wikimedia.org/g/analytics/refinery analytics/refinery repository].

==How to deploy==
The refinery now uses [[Scap3|Scap]] so the deployments are as simple as:

*IMPORTANT: wait about 5 minutes if you just did a refinery-source build.  There's a cron that creates the git-lfs symlinks and scap will fail if those are missing.  And if it does fail, you'll have problems later with the repository on the deploy target (eg. stat1007's /srv/deployment/analytics/refinery will be in an inconsistent state and you'll need to git lfs fetch --all or similar)
*First ssh: &lt;code&gt;ssh deployment.eqiad.wmnet&lt;/code&gt;
*Make sure scap config is up to date:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cd /srv/deployment/analytics/refinery/scap

git pull
&lt;/syntaxhighlight&gt;

*Tell the &lt;code&gt;#wikimedia-analytics&lt;/code&gt; and &lt;code&gt;#wikimedia-operations&lt;/code&gt; IRC channels that you are deploying (e.g., &lt;code&gt;!log Deploying Refinery at 37a2ddf for change 1126977 / T388654 tlwikisource to allowlist&lt;/code&gt; ; modify this to fit the commit hash, patch(es), tickets and shrink text if needed)
** Use &lt;code&gt;git rev-parse --short HEAD&lt;/code&gt; to get the commit ID
*Create a screen/tmux session to prevent network failures from ruining your deployment.
*Run:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
# step 1
cd /srv/deployment/analytics/refinery

# step 2
git pull

# step 3
scap deploy -e hadoop-test &quot;Regular analytics weekly train TEST [analytics/refinery@$(git rev-parse --short HEAD)]&quot;
# wait until this completes


# step 4
scap deploy &quot;Regular analytics weekly train [analytics/refinery@$(git rev-parse --short HEAD)]&quot;
# wait until this completes

# step 5
scap deploy -e thin &quot;Regular analytics weekly train THIN [analytics/refinery@$(git rev-parse --short HEAD)]&quot;
# wait until this completes

&lt;/syntaxhighlight&gt;


&lt;code&gt;scap deploy -e hadoop-test&lt;/code&gt; will deploy to a testing environment (see &lt;code&gt;environments/hadoop-test/targets&lt;/code&gt; in the &lt;code&gt;scap&lt;/code&gt; directory on the deployment server, and see below; you should also follow the &quot;After the deployment&quot; bullet point a couple points down on the corresponding test host).

The plain &lt;code&gt;scap deploy&lt;/code&gt; will deploy to the canary host first (&lt;code&gt;stat1008&lt;/code&gt;, see the &lt;code&gt;target-canary&lt;/code&gt; file from the &lt;code&gt;scap&lt;/code&gt; directory on the deployment server) and then if everything goes fine it will ask you to proceed with the rest of the hosts.

&lt;code&gt;scap deploy -e thin&lt;/code&gt; will deploy to clouddumps hosts as described below (see &lt;code&gt;environments/thin/targets&lt;/code&gt; in the &lt;code&gt;scap&lt;/code&gt; directory on the deployment server).

If you want to double check the status of a deployment, you can run &lt;code&gt;scap deploy-log&lt;/code&gt; in another terminal and check what is happening.

Please note that Scap will create another copy of the repository after each deployment, and it will use symlinks to switch versions.

*'''Make sure''' that all the deployment are finished, and went fine. Rollback in case of fireworks.
**'''WARNING''': while the rollback process is generally safe, a rollback following git lfs related issues might leave the refinery working directory in a stale state. This in turn will prevent rolled back deps to deployed. If unsure on how to clean up the working directory, do reach out to the team in #wikmedia-analytics (IRC)
*'''After the scap deployment, an HDFS deployment is also needed:'''
** ssh into [[an-launcher1003|&lt;code&gt;an-launcher1003&lt;/code&gt;]] and/or &lt;code&gt;an-test-coord1001&lt;/code&gt; (the latter is for the test cluster).
** Make sure to wait until the entire scap deployment is completed before ssh to &lt;code&gt;an-launcher1003&lt;/code&gt; (or &lt;code&gt;an-test-cord1001&lt;/code&gt; for the test cluster). If you cd into the refinery directory before the symlinks change, you'll still be seeing the old git log and be very confused.
** Create a screen or a tmux session to prevent network failures to interfere with the execution of the following command.
** Run:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
# step 1 - reach deployment folder
cd /srv/deployment/analytics/refinery

# step 2 - check last commit message
sudo -u analytics-deploy /usr/bin/git log -n 1

# step 3 - deploy code onto HDFS
/srv/deployment/analytics/refinery/bin/refinery-deploy-to-hdfs --verbose --no-dry-run

&lt;/syntaxhighlight&gt;
*Finally, consider changing any documentation that needs to be updated. This may include: [[Analytics/Data/Webrequest]], [[Analytics/Data/Pageview_hourly]], [[m:Research:Page_view|Research:Page_view]] (and its sub-pages).

{{Notice|Remember to log onto the #wikimedia-analytics and #wikimedia-operations IRC channel upon successful deploy, for instance: &lt;code&gt;!log Deployed refinery using scap, then deployed onto hdfs&lt;/code&gt;.)}}

=== Deploying to clouddumps* hosts ===
We deploy refinery to labstore separately from our main deploy targets, avoiding git-fat objects to save disk space.  The clouddumps nodes need the hdfs-rsync script to pull regularly public data from Hadoop. The scap &quot;thin&quot; environment deploys and clouddumps* hosts right now, as described above.

&lt;code&gt;scap deploy -e thin&lt;/code&gt;

=== Deploying to Hadoop test ===
See the general instructions above. But in case you need to check something on Hadoop test first, see the instructions above and note the use of &lt;code&gt;scap deploy -e hadoop-test&lt;/code&gt; .

Then you need to upload refinery to HDFS from an-test-coord1001 (see above for the &lt;code&gt;refinery-deploy-to-hdfs&lt;/code&gt; command).

==How to manually modify tables==
Some changes in Refinery might need manual operations on tables. For example, a change in a schema may require to drop a table, recreate it, and repair it before deploying. 

Before doing any change from an an-launcher server, you need to deploy following the documentation. After the deployment, the an-launcher will have the new code ready. 

To modify tables in the &lt;code&gt;wmf&lt;/code&gt; schema you need to work as the &lt;code&gt;analytics&lt;/code&gt; user. 
From an &lt;/code&gt;an-launcher&lt;/code&gt; server:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
kinit

sudo -u analytics kerberos-run-command analytics spark3-sql --master yarn --executor-memory 8G --executor-cores 4 --driver-memory 2G --conf spark.dynamicAllocation.maxExecutors=64
&lt;/syntaxhighlight&gt;

If you need to run a DDL, it's often:

&lt;syntaxhighlight lang&quot;bash&quot;&gt;
sudo -u analytics kerberos-run-command analytics hive -f /srv/deployment/analytics/refinery/hql/&lt;needed_file&gt;.hql --database &lt;database_name&gt;
&lt;/syntaxhighlight&gt;

To repair a table:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sudo -u analytics kerberos-run-command analytics hive -e &quot;MSCK REPAIR TABLE &lt;database&gt;.&lt;table&gt;;&quot;
&lt;/syntaxhighlight&gt;

[[Category:Data platform]]
[[Category:Data platform systems]]</text>
      <sha1>p8pcok5b3e3fhlapmh7pwympba9puj3</sha1>
    </revision>
    <revision>
      <id>2435511</id>
      <parentid>2435510</parentid>
      <timestamp>2026-07-12T01:05:14Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix my prior</comment>
      <origin>2435511</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6762" sha1="bg5pwp5p6d48dz7phufna2x8tv33erz" xml:space="preserve">Refinery is the software infrastructure that is used on the [[Analytics/Cluster|Analytics Cluster]]. The source code is in the [https://gerrit.wikimedia.org/g/analytics/refinery analytics/refinery repository].

==How to deploy==
The refinery now uses [[Scap3|Scap]] so the deployments are as simple as:

*IMPORTANT: wait about 5 minutes if you just did a refinery-source build.  There's a cron that creates the git-lfs symlinks and scap will fail if those are missing.  And if it does fail, you'll have problems later with the repository on the deploy target (eg. stat1007's /srv/deployment/analytics/refinery will be in an inconsistent state and you'll need to git lfs fetch --all or similar)
*First ssh: &lt;code&gt;ssh deployment.eqiad.wmnet&lt;/code&gt;
*Make sure scap config is up to date:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cd /srv/deployment/analytics/refinery/scap

git pull
&lt;/syntaxhighlight&gt;

*Tell the &lt;code&gt;#wikimedia-analytics&lt;/code&gt; and &lt;code&gt;#wikimedia-operations&lt;/code&gt; IRC channels that you are deploying (e.g., &lt;code&gt;!log Deploying Refinery at 37a2ddf for change 1126977 / T388654 tlwikisource to allowlist&lt;/code&gt; ; modify this to fit the commit hash, patch(es), tickets and shrink text if needed)
** Use &lt;code&gt;git rev-parse --short HEAD&lt;/code&gt; to get the commit ID
*Create a screen/tmux session to prevent network failures from ruining your deployment.
*Run:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
# step 1
cd /srv/deployment/analytics/refinery

# step 2
git pull

# step 3
scap deploy -e hadoop-test &quot;Regular analytics weekly train TEST [analytics/refinery@$(git rev-parse --short HEAD)]&quot;
# wait until this completes


# step 4
scap deploy &quot;Regular analytics weekly train [analytics/refinery@$(git rev-parse --short HEAD)]&quot;
# wait until this completes

# step 5
scap deploy -e thin &quot;Regular analytics weekly train THIN [analytics/refinery@$(git rev-parse --short HEAD)]&quot;
# wait until this completes

&lt;/syntaxhighlight&gt;


&lt;code&gt;scap deploy -e hadoop-test&lt;/code&gt; will deploy to a testing environment (see &lt;code&gt;environments/hadoop-test/targets&lt;/code&gt; in the &lt;code&gt;scap&lt;/code&gt; directory on the deployment server, and see below; you should also follow the &quot;After the deployment&quot; bullet point a couple points down on the corresponding test host).

The plain &lt;code&gt;scap deploy&lt;/code&gt; will deploy to the canary host first (&lt;code&gt;stat1008&lt;/code&gt;, see the &lt;code&gt;target-canary&lt;/code&gt; file from the &lt;code&gt;scap&lt;/code&gt; directory on the deployment server) and then if everything goes fine it will ask you to proceed with the rest of the hosts.

&lt;code&gt;scap deploy -e thin&lt;/code&gt; will deploy to clouddumps hosts as described below (see &lt;code&gt;environments/thin/targets&lt;/code&gt; in the &lt;code&gt;scap&lt;/code&gt; directory on the deployment server).

If you want to double check the status of a deployment, you can run &lt;code&gt;scap deploy-log&lt;/code&gt; in another terminal and check what is happening.

Please note that Scap will create another copy of the repository after each deployment, and it will use symlinks to switch versions.

*'''Make sure''' that all the deployment are finished, and went fine. Rollback in case of fireworks.
**'''WARNING''': while the rollback process is generally safe, a rollback following git lfs related issues might leave the refinery working directory in a stale state. This in turn will prevent rolled back deps to deployed. If unsure on how to clean up the working directory, do reach out to the team in #wikmedia-analytics (IRC)
*'''After the scap deployment, an HDFS deployment is also needed:'''
** ssh into [[an-launcher1003|&lt;code&gt;an-launcher1003&lt;/code&gt;]] and/or &lt;code&gt;an-test-coord1001&lt;/code&gt; (the latter is for the test cluster).
** Make sure to wait until the entire scap deployment is completed before ssh to &lt;code&gt;an-launcher1003&lt;/code&gt; (or &lt;code&gt;an-test-cord1001&lt;/code&gt; for the test cluster). If you cd into the refinery directory before the symlinks change, you'll still be seeing the old git log and be very confused.
** Create a screen or a tmux session to prevent network failures to interfere with the execution of the following command.
** Run:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
# step 1 - reach deployment folder
cd /srv/deployment/analytics/refinery

# step 2 - check last commit message
sudo -u analytics-deploy /usr/bin/git log -n 1

# step 3 - deploy code onto HDFS
/srv/deployment/analytics/refinery/bin/refinery-deploy-to-hdfs --verbose --no-dry-run

&lt;/syntaxhighlight&gt;
*Finally, consider changing any documentation that needs to be updated. This may include: [[Analytics/Data/Webrequest]], [[Analytics/Data/Pageview_hourly]], [[m:Research:Page_view|Research:Page_view]] (and its sub-pages).

{{Notice|Remember to log onto the #wikimedia-analytics and #wikimedia-operations IRC channel upon successful deploy, for instance: &lt;code&gt;!log Deployed refinery using scap, then deployed onto hdfs&lt;/code&gt;.)}}

=== Deploying to clouddumps* hosts ===
We deploy refinery to labstore separately from our main deploy targets, avoiding git-fat objects to save disk space.  The clouddumps nodes need the hdfs-rsync script to pull regularly public data from Hadoop. The scap &quot;thin&quot; environment deploys and clouddumps* hosts right now, as described above.

&lt;code&gt;scap deploy -e thin&lt;/code&gt;

=== Deploying to Hadoop test ===
See the general instructions above. But in case you need to check something on Hadoop test first, see the instructions above and note the use of &lt;code&gt;scap deploy -e hadoop-test&lt;/code&gt; .

Then you need to upload refinery to HDFS from an-test-coord1001 (see above for the &lt;code&gt;refinery-deploy-to-hdfs&lt;/code&gt; command).

==How to manually modify tables==
Some changes in Refinery might need manual operations on tables. For example, a change in a schema may require to drop a table, recreate it, and repair it before deploying. 

Before doing any change from an an-launcher server, you need to deploy following the documentation. After the deployment, the an-launcher will have the new code ready. 

To modify tables in the &lt;code&gt;wmf&lt;/code&gt; schema you need to work as the &lt;code&gt;analytics&lt;/code&gt; user. 
From an &lt;/code&gt;an-launcher&lt;/code&gt; server:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
kinit

sudo -u analytics kerberos-run-command analytics spark3-sql --master yarn --executor-memory 8G --executor-cores 4 --driver-memory 2G --conf spark.dynamicAllocation.maxExecutors=64
&lt;/syntaxhighlight&gt;

If you need to run a DDL, it's often:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sudo -u analytics kerberos-run-command analytics hive -f /srv/deployment/analytics/refinery/hql/&lt;needed_file&gt;.hql --database &lt;database_name&gt;
&lt;/syntaxhighlight&gt;

To repair a table:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sudo -u analytics kerberos-run-command analytics hive -e &quot;MSCK REPAIR TABLE &lt;database&gt;.&lt;table&gt;;&quot;
&lt;/syntaxhighlight&gt;

[[Category:Data platform]]
[[Category:Data platform systems]]</text>
      <sha1>bg5pwp5p6d48dz7phufna2x8tv33erz</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Paws/SAL</title>
    <ns>498</ns>
    <id>55577</id>
    <revision>
      <id>2435491</id>
      <parentid>2433382</parentid>
      <timestamp>2026-07-11T12:55:25Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>dcaro: Out of space on nfs, looking</comment>
      <origin>2435491</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="56120" sha1="skvjfbsanrwxqpo2rn9w6abgs8xunh0" xml:space="preserve">=== 2026-07-11 ===
* 12:55 dcaro: Out of space on nfs, looking

=== 2026-07-06 ===
* 16:42 andrewbogott: running &quot;find . -name ubuntu24.img -delete&quot; and &quot;find . -name ubuntu22.img -delete&quot; in /srv/paws/project/paws/userhomes

=== 2026-07-02 ===
* 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])

=== 2026-06-29 ===
* 13:15 andrewbogott: rm -rf /srv/paws/files-to-remove/* on paws-nfs-2

=== 2026-06-23 ===
* 19:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2026-06-22 ===
* 15:43 andrewbogott: rm -rf /srv/paws/files-to-remove/* for [[phab:T429578|T429578]]

=== 2026-05-28 ===
* 17:25 andrewbogott: switched proxies to new paws release featuring pywikibot 1.30.0

=== 2026-05-07 ===
* 21:29 andrewbogott: rebooting all worker nodes to pick up modprobe changes

=== 2026-04-28 ===
* 09:46 volans: killed 22 zombies pods stuck in Terminating status since many days and with no running container/process

=== 2026-03-13 ===
* 20:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1)
* 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm paws-127c-uwce57bvcgrt-master-0 (cluster eqiad1)
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm paws-127c-uwce57bvcgrt-master-0 (cluster eqiad1)

=== 2026-03-09 ===
* 08:31 godog: archive outdated https://gitlab.wikimedia.org/repos/cloud/paws in favor of github repo

=== 2025-12-24 ===
* 17:21 andrewbogott: deleting everything in /srv/paws/files-to-remove [[phab:T396051|T396051]]

=== 2025-12-23 ===
* 10:56 dhinus: hard-reboot paws-127c-uwce57bvcgrt-node-1 (reporting NotReady in kubectl)

=== 2025-12-02 ===
* 20:50 andrewbogott: stop/starting paws-127b-rpchztfjt2jb-node-1 to fix mtu mismatch
* 20:00 andrewbogott: rebooting paws-127b-rpchztfjt2jb-master-0.paws to aid with host draining

=== 2025-11-19 ===
* 09:32 dcaro: reboot worker 2 and 4
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-2 (cluster eqiad1, project paws)
* 09:31 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-2 (cluster eqiad1, project paws)
* 09:31 dcaro: reboot worker 1
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)

=== 2025-10-08 ===
* 12:35 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]]

=== 2025-10-02 ===
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:27 dcaro: reboot node-4 ([[phab:T406191|T406191]])

=== 2025-09-26 ===
* 09:17 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)

=== 2025-09-25 ===
* 16:56 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)
* 16:56 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)
* 16:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-09-22 ===
* 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1, project paws)
* 08:54 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1, project paws)

=== 2025-09-18 ===
* 19:37 andrewbogott: replaced previous deployment with new 127c deployment, supporting ssh access to VMs for [[phab:T405023|T405023]]
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 18:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-09-15 ===
* 17:19 andrewbogott: hard rebooting paws worker nodes one by one
* 12:56 dcaro: restarting the k8s workers 1, 2 and 4
* 12:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127b-rpchztfjt2jb-node-4 (cluster eqiad1, project paws)
* 12:56 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127b-rpchztfjt2jb-node-4 (cluster eqiad1, project paws)
* 12:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127b-rpchztfjt2jb-node-2 (cluster eqiad1, project paws)
* 12:54 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127b-rpchztfjt2jb-node-2 (cluster eqiad1, project paws)
* 12:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127b-rpchztfjt2jb-node-1 (cluster eqiad1, project paws)
* 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127b-rpchztfjt2jb-node-1 (cluster eqiad1, project paws)

=== 2025-09-10 ===
* 08:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'volans' in role 'member'
* 08:16 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'volans' in role 'member'

=== 2025-09-09 ===
* 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 12:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0)
* 12:16 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot
* 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0)
* 12:15 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot
* 12:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-08-04 ===
* 14:23 andrewbogott: service restored
* 14:19 andrewbogott: restarting all paws k8s nodes
* 14:19 andrewbogott: extended paws-nfs volume ran e2fsck and resize2fs on paws-nfs-1 and rebooted
* 14:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 14:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 14:05 andrewbogott: stopping all nodes to clear the way for NFS maintenance
* 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]])
* 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]])

=== 2025-07-29 ===
* 20:19 andrewbogott: removing Alex Monk from project in response to his email request
* 09:00 taavi: deploy pywikibot 10.2.0 update [[phab:T394614|T394614]]

=== 2025-07-25 ===
* 14:26 andrewbogott: moving all home dirs with files named 'android' into /srv/paws/files-to-remove/

=== 2025-07-24 ===
* 12:40 andrewbogott: purging /srv/paws/files-to-remove

=== 2025-07-13 ===
* 03:31 andrewbogott: rebooted all paws worker nodes, which may or may not have fixed the service (it was down before and is back now!)

=== 2025-07-03 ===
* 12:53 andrewbogott: moved all notebooks with Android binaries to /srv/paws/files-to-remove/

=== 2025-06-30 ===
* 09:42 dcaro: deleted the collected files-to-remove due to 100% capacity on nfs

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2024-06-20 ===
* 11:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0)
* 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs

=== 2024-06-18 ===
* 20:46 andrewbogott: rebuilt magnum cluster using g4/ovs flavors

=== 2024-05-06 ===
* 17:15 Rook: upgrade pywikibot [[phab:T364188|T364188]]

=== 2024-04-29 ===
* 17:54 Rook: upgrade pywikibot [[phab:T363131|T363131]]
* 17:54 Rook: jupyterlab to 4.1.8 [[phab:T363596|T363596]]
* 16:50 Rook: k8s to 1.26 [[phab:T326985|T326985]]

=== 2024-03-25 ===
* 15:59 Rook: upgrade jupyter chart. New cluster [[phab:T360643|T360643]]
* 13:01 Rook: use upstream jupyter-rsession-proxy [[phab:T360800|T360800]]

=== 2024-03-15 ===
* 16:16 Rook: upgrade jupyterlab [[phab:T360193|T360193]]

=== 2024-03-09 ===
* 16:06 Rook: increase number of worker nodes [[phab:T359747|T359747]]

=== 2024-03-08 ===
* 12:33 Rook: upgrade pywikibot [[phab:T359616|T359616]]
* 11:49 Rook: upgrade jupyterlab version [[phab:T359588|T359588]]

=== 2024-03-07 ===
* 21:01 Rook: increase worker count to manage outreachy load [[phab:T359591|T359591]]

=== 2024-03-06 ===
* 12:07 Rook: increase capacity for outreachy [[phab:T359316|T359316]]

=== 2024-03-05 ===
* 13:25 Rook: remove jupyter-dash [[phab:T358621|T358621]]

=== 2024-03-04 ===
* 13:57 Rook: add wikibase-cli [[phab:T358649|T358649]]

=== 2024-02-26 ===
* 16:09 Rook: remove nbextension moving away from notebook interface [[phab:T312234|T312234]]

=== 2024-02-21 ===
* 16:07 Rook: increase prometheus retention [[phab:T357786|T357786]]

=== 2024-02-20 ===
* 16:17 Rook: upgrade jupyterlab [[phab:T357990|T357990]]

=== 2024-02-15 ===
* 18:33 Rook: update openresty [[phab:T357698|T357698]]
* 13:59 Rook: upgrade jupyterlab to 4.1.1 [[phab:T357027|T357027]]

=== 2024-02-12 ===
* 15:56 Rook: upgrade OpenRefine [[phab:T356448|T356448]]

=== 2024-02-09 ===
* 17:48 Rook: jupyterlab upgraded to 4.1.0 [[phab:T357027|T357027]]

=== 2024-02-01 ===
* 20:17 Rook: prometheus and kube-state-metrics internal to cluster [[phab:T355179|T355179]]

=== 2024-01-29 ===
* 12:46 Rook: update jupyerlab [[phab:T355890|T355890]]

=== 2024-01-26 ===
* 15:25 Rook: update to allow for s3 tofu state storage in codfw1dev [[phab:T355543|T355543]]

=== 2024-01-24 ===
* 12:56 Rook: Remove 123-11 cluster [[phab:T355785|T355785]]

=== 2024-01-18 ===
* 08:05 Rook: upgrade rstudio-server [[phab:T355288|T355288]]

=== 2024-01-12 ===
* 14:00 Rook: removed paws-123-10 cluster [[phab:T354946|T354946]]

=== 2023-12-08 ===
* 14:41 Rook: upgrade OpenRefine [[phab:T353021|T353021]]

=== 2023-12-06 ===
* 14:28 Rook: pywikibot to 8.6 [[phab:T352794|T352794]]

=== 2023-11-27 ===
* 12:33 Rook: jupyterlab to 4.0.9 [[phab:T351726|T351726]]

=== 2023-11-15 ===
* 19:46 Rook: move to opentofu [[phab:T351249|T351249]]

=== 2023-11-13 ===
* 18:17 Rook: remove old cluster [[phab:T350875|T350875]]
* 13:21 Rook: pwb version bump [[phab:T351015|T351015]]

=== 2023-11-09 ===
* 14:34 Rook: updated ingress-nginx [[phab:T347506|T347506]]

=== 2023-11-06 ===
* 17:10 Rook: pywikibot to 8.5.0 [[phab:T350552|T350552]]

=== 2023-11-03 ===
* 09:27 Rook: bump jupyterlab version [[phab:T350459|T350459]]

=== 2023-10-24 ===
* 13:03 Rook: removed old cluster [[phab:T349551|T349551]]

=== 2023-10-23 ===
* 18:35 Rook: deploy new cluster/jupyterhub chart [[phab:T349545|T349545]]

=== 2023-10-19 ===
* 15:37 Rook: Bump jupyterlab version [[phab:T349203|T349203]]
* 14:53 Rook: Bump urllib3 from 1.26.17 to 1.26.18
* 12:59 Rook: Bump urllib3 from 1.26.16 to 1.26.17

=== 2023-10-18 ===
* 12:31 Rook: Jupyterlab to 3.0.6 [[phab:T347108|T347108]]
* 11:01 Rook: update OpenRefine [[phab:T348464|T348464]]

=== 2023-10-09 ===
* 15:50 Rook: bump pwb version [[phab:T348372|T348372]]

=== 2023-09-29 ===
* 09:58 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:56 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-09-21 ===
* 14:24 Rook: bump pwb version [[phab:T346912|T346912]]

=== 2023-09-05 ===
* 12:19 Rook: Reduce memory request for single user container [[phab:T345467|T345467]]

=== 2023-09-01 ===
* 18:45 Rook: increased worker node count to as all memory was requested [[phab:T345462|T345462]]

=== 2023-08-31 ===
* 16:51 taavi: restart uwsgi-toolsdb-replica-cnf-web.service on paws-nfs-2, a required dependency was installed only after last restart which made the service fail
* 13:52 Rook: pwb to version 8.3.2 [[phab:T345192|T345192]]

=== 2023-08-30 ===
* 12:39 Rook: Upgrade notebook allowing i18n support [[phab:T121478|T121478]]

=== 2023-08-28 ===
* 13:37 Rook: upgrade node [[phab:T341716|T341716]]

=== 2023-08-22 ===
* 18:06 Rook: bump jupyterlab version [[phab:T344265|T344265]]

=== 2023-08-21 ===
* 20:03 Rook: pywikibot version bump [[phab:T344493|T344493]]

=== 2023-08-18 ===
* 14:07 Rook: unrar added

=== 2023-08-15 ===
* 14:08 Rook: [[phab:T343116|T343116]] update helm chart, jupyterlab, jupyterhub, notebook. Dropping sparql to allow for update.

=== 2023-07-28 ===
* 12:26 Rook: pwb version bump [[phab:T342852|T342852]]

=== 2023-07-17 ===
* 18:33 Rook: update OpenRefine version [[phab:T341985|T341985]]

=== 2023-07-13 ===
* 14:06 Rook: terraform fmt [[phab:T341784|T341784]]
* 12:47 Rook: move db to pvc [[phab:T341132|T341132]]

=== 2023-07-12 ===
* 15:22 Rook: paws deploying from tf in codfw1dev [[phab:T340657|T340657]]

=== 2023-07-10 ===
* 11:18 Rook: shut down systems associated with [[phab:T341457|T341457]] should only be redirect page.
* 11:00 Rook: removed paws-db-backup-1 [[phab:T340054|T340054]]

=== 2023-06-30 ===
* 18:51 Rook: https://wikitech.wikimedia.org/wiki/PAWS/Admin updated to describe blue green deployment method using terraform
* 18:48 Rook: terraform working in eqiad1 [[phab:T340836|T340836]]
* 18:18 Rook: bounced control node as it was getting disk pressure

=== 2023-06-29 ===
* 13:52 Rook: [[phab:T340698|T340698]] OpenRefine version bump

=== 2023-06-13 ===
* 18:16 Rook: Rolling jupyterlab back to last known working version [[phab:T338981|T338981]]

=== 2023-06-12 ===
* 16:02 Rook: updated jupyterlab [[phab:T338337|T338337]]

=== 2023-06-07 ===
* 10:30 Rook: upgrade jupyterlab [[phab:T324002|T324002]]

=== 2023-05-15 ===
* 12:00 Rook: implemented limits to storage. Cleared about half the storage used [[phab:T327936|T327936]]

=== 2023-05-03 ===
* 12:26 Rook: upgrade jupyterhub {{Gerrit|fb5c7a33b4f6040b20ae19f4d3c368d3e3c0df29}} [[phab:T335810|T335810]]

=== 2023-04-24 ===
* 13:28 Rook: pywikibot version bump {{Gerrit|546696214f1a5e50e297241d22042a7cee870e11}} [[phab:T335237|T335237]]
* 11:37 andrewbogott: rebooting paws-k8s-haproxy-2, paws-acme-chief-01, paws-k8s-haproxy-1 -- their load numbers are through the roof for no obvious reason

=== 2023-04-17 ===
* 11:13 Rook: robots.txt to nbserve {{Gerrit|1fbc7865aad1ea592d9a852dda3bb3386fc1f29c}}

=== 2023-04-13 ===
* 21:19 Rook: revert jupyterlab upgrade, causing notebooks to not load [[phab:T324002|T324002]]

=== 2023-04-12 ===
* 14:42 Rook: upgrade jupyterhub and jupyterlab {{Gerrit|c3450668212846ef1430941d17f9890274247a4d}} [[phab:T324002|T324002]]
* 10:51 Rook: remove focal from R {{Gerrit|7a88b67d990035eb912c5399f5ea3deeb4b75f37}} [[phab:T331363|T331363]]

=== 2023-04-10 ===
* 12:40 Rook: openrefine version bump {{Gerrit|771250537d42d16bed997c3bd1f7f500512e634b}} [[phab:T334145|T334145]]

=== 2023-04-03 ===
* 14:31 Rook: pywikibot version bump {{Gerrit|c385d673b9a9fedd29f7ef6e586f527a93ae2f50}} [[phab:T333482|T333482]]

=== 2023-03-27 ===
* 19:41 Rook: Upgrade Wikimedia Commons extension for OpenRefine on PAWS to version 0.1.1 {{Gerrit|4b10d3f0d272d7dfb88022eb123232ff9b2f202f}} [[phab:T332721|T332721]]
* 17:50 Rook: bump pwb version {{Gerrit|7c4951863ae5a356c5b978f41b5eb4b465a72138}} [[phab:T333067|T333067]]

=== 2023-03-20 ===
* 13:12 Rook: move paws to k8s 1.23 {{Gerrit|b1d0daeb20cfeb6981653d65e44cf6175eded539}} [[phab:T328489|T328489]]

=== 2023-03-16 ===
* 13:13 Rook: pywikibot version bump {{Gerrit|6de17dbcbeba8da9e79cfae637945f7da4afc851}}

=== 2023-03-15 ===
* 16:46 Rook: Install Wikimedia Commons extension for OpenRefine {{Gerrit|aaae2fd0fb198fcf37efaba3d910fd0306c6763b}} Install Wikimedia Commons extension for OpenRefine
* 16:08 Rook: openrefine version bump {{Gerrit|9047213e25cd0a5ecc342ceb62f1a962c303d261}} [[phab:T331747|T331747]]

=== 2023-03-14 ===
* 14:21 Rook: NFS cutover {{Gerrit|0ab68a892bf40c6237d8edf5d9c23f2cbc430da0}} [[phab:T331056|T331056]]

=== 2023-03-13 ===
* 10:40 Rook: Restructure paws away from special networking (Change paws domain name) {{Gerrit|df16f355de3856c9ef7ef72ea4ae86dc9080723f}} [[phab:T328842|T328842]]

=== 2023-02-20 ===
* 21:08 Rook: upgrade openrefine to 3.7.0 {{Gerrit|adc41fc9ce208197589adae938bbc8374ae80c5b}} [[phab:T329835|T329835]]

=== 2023-02-18 ===
* 20:48 Rook: revert to installing pywikibot from git {{Gerrit|a22cd26b16dc256f505e70d30c3d10e038da747d}} [[phab:T326512|T326512]]

=== 2023-02-16 ===
* 12:58 Rook: moving to new cluster. Old one was restarting hub and couldn't find all of its nodes

=== 2023-02-14 ===
* 13:27 Rook: Bump oauthlib {{Gerrit|97f241bacff7af60cebfffa0d8eb945c7545577d}}
* 08:15 taavi: empty profile::wmcs::paws::control_nodes hiera key to bring PAWS back up ([[phab:T329581|T329581]]), it contained the hostnames of the old kubeadm backed cluster which should be cleaned up properly in [[phab:T327674|T327674]]

=== 2023-02-13 ===
* 12:20 Rook: pywikibot to version 8 {{Gerrit|16f07db23b6e380a15297605cbbb532f312a3246}} [[phab:T326512|T326512]]

=== 2023-01-31 ===
* 12:24 Rook: cluster upgraded to k8s 1.22 {{Gerrit|360b1238e85a7b3613db88a4919e87801ae69321}} [[phab:T327264|T327264]]

=== 2023-01-30 ===
* 13:51 Rook: Set 1 to = 1 for nfs mounts and variables {{Gerrit|09b036eb9dab09df39e9064cad4d18bc0db1b763}} [[phab:T326675|T326675]]
* 11:53 Rook: updated ingress-nginx to allow larger file (more than 800K) uploads [[phab:T328168|T328168]]

=== 2023-01-25 ===
* 15:00 Rook: deploy nginx config as configmap {{Gerrit|1b418695b7fc04050e4ab278642f2f26875eddef}} [[phab:T322205|T322205]]

=== 2023-01-24 ===
* 18:52 Rook: remove unused lighthttpd.conf {{Gerrit|0c92944008e9b17426c7bd08e25946b0a175f8d9}} [[phab:T322064|T322064]]
* 12:44 Rook: removing unneeded lines {{Gerrit|897e64e68613f201bfacbd4fe3f1b25be0c219d6}} [[phab:T322742|T322742]]
* 10:54 Rook: bump certifi version {{Gerrit|ee13069fcf84bfa9fa2fedc519bea9a9639c5298}} [[phab:T327764|T327764]]

=== 2023-01-23 ===
* 16:21 Rook: remove automatic update for pywikibot {{Gerrit|029126aea467129de4d6aeb4981c938adc0a48e3}} [[phab:T327668|T327668]]
* 13:09 Rook: remove missed dbProxy entries {{Gerrit|7c52b75f8e00a2af2fd7d75897a959ab841e4d70}} [[phab:T327414|T327414]]

=== 2023-01-19 ===
* 15:55 Rook: cpu limit for renderer {{Gerrit|c48dec0e2d7555169c1faf7e07bf7032ecaa5785}} [[phab:T327306|T327306]]

=== 2023-01-17 ===
* 15:24 Rook: move to magnum {{Gerrit|0eae9000de69fd4d490cb4d87b8c724d11a21350}} [[phab:T326554|T326554]]

=== 2023-01-12 ===
* 14:09 Rook: Variables for nbserve and renderer resource requests {{Gerrit|4eada3f1769a0d48c3bbd1d9a52e3ee8b95e907e}} [[phab:T326723|T326723]]

=== 2023-01-11 ===
* 12:12 Rook: variablize nfs entries for paws-dev {{Gerrit|78c50a9b3fb836b473351bdabb20607199234262}} [[phab:T326631|T326631]]

=== 2023-01-10 ===
* 17:37 Rook: setup dns for paws-dev {{Gerrit|837395a6695d11ae9026df31dd57b25df02102b1}} [[phab:T326629|T326629]]

=== 2023-01-04 ===
* 12:50 Rook: ingress options for ingress-nginx {{Gerrit|4b753ffd22bac5c8ed5085feb4d4e93de2e31e61}} [[phab:T325746|T325746]]

=== 2023-01-03 ===
* 18:45 Rook: update admin users {{Gerrit|ad540ad96af1217d450f8a25cb5427b2e2ad6afd}} [[phab:T326165|T326165]]
* 18:45 Rook: upgrade jupyterhub  {{Gerrit|52027e5fe05e84f69852e5b6ff5fc965fa4ebfc5}} [[phab:T325812|T325812]]

=== 2022-12-16 ===
* 16:08 dcaro: removing coe cluster rook3 ([[phab:T325373|T325373]])

=== 2022-12-13 ===
* 12:48 Rook: bump pywikibot version {{Gerrit|fea0a1a38c8d3a5e2cd29f8fa854664dd7cb6643}} [[phab:T324404|T324404]]

=== 2022-12-05 ===
* 19:40 Rook: Do not load index files automatically {{Gerrit|cd85e584824c983ac5eef6fe1a6010d69c4c8003}} [[phab:T296870|T296870]]

=== 2022-12-01 ===
* 08:30 taavi: root@paws-k8s-control-1:~# for cert in etcd-server etcd-peer etcd-healthcheck-client; do kubeadm certs renew $cert; done # [[phab:T324178|T324178]]

=== 2022-11-28 ===
* 15:43 Rook: drop hostpath [[phab:T321886|T321886]] {{Gerrit|ce2b1f59101e5eb978c2c04d138e704f1363e9fb}}

=== 2022-11-02 ===
* 13:17 Rook: [[phab:T322063|T322063]] Adding commentary regarding nginx rewrites {{Gerrit|b11757be2cd086f2846c2ae154a04a45aa67a7af}}

=== 2022-10-31 ===
* 11:52 Rook: jupyterhub to 3.0 [[phab:T318271|T318271]] {{Gerrit|dc95efa69e1e9887daaa6320dcbd4acc94f624de}}

=== 2022-10-27 ===
* 09:10 Rook: [[phab:T319368|T319368]] openrefine bump {{Gerrit|fe94bc1be2eaca1f87e16f2e8ef685aba46cc4d9}}

=== 2022-10-24 ===
* 15:06 Rook: [[phab:T318273|T318273]] Upgrade jupyterlab {{Gerrit|b342570b6bf55ebc5b3965f7d1fffa5cb4d06157}}

=== 2022-10-20 ===
* 09:57 Rook: autoscaling for renderer deployment [[phab:T320776|T320776]]   {{Gerrit|910e3ca0dbf2a2b9019ff3d86520565e4548d778}}

=== 2022-10-17 ===
* 10:04 Rook: update favicon to paws logo [[phab:T198012|T198012]] {{Gerrit|4019bd87bc9ec588a35944325569177962ccc24d}}
* 05:09 Rook: readme for paws-hub {{Gerrit|1d125c2b88f1c479eb9e1a9250716b49fd4c1e2d}} [[phab:T308949|T308949]]
* 04:40 Rook: update favicon on hub image [[phab:T198012|T198012]] {{Gerrit|146effd07d2adda41d1c1dc9ff5827fae0012ef4}}

=== 2022-10-10 ===
* 17:02 Rook: Bump pywikibot to 7.7.1 {{Gerrit|24d0e2223175024bd11465decbf0a1087c1cc2c3}} [[phab:T320432|T320432]]
* 12:13 Rook: [[phab:T318279|T318279]] Tidy R config files {{Gerrit|c220d165ea779946fa7a3ff98cf806f023e46109}}
* 05:58 Rook: Upgrade IRkernel {{Gerrit|fd62fefdcd42ced29f30be56f82f14ed765fe780}} [[phab:T318275|T318275]]

=== 2022-10-05 ===
* 10:40 dcaro: extended the /srv volumes for both prometheus nodes to 15G to give some space for prometheus to shuffle data (the tsdb size is set to 10G, but it needs a bit more space than that)
* 10:18 arturo: aborrero@paws-k8s-control-1:~$ sudo -i kubectl -n prod rollout restart deployment/proxy ([[phab:T319366|T319366]])
* 10:02 dcaro: checking the prometheus instance for mounts from labstore
* 10:01 dcaro: checking db-backup for labstore mounts (it should have according to hiera)
* 10:00 dcaro: note that ingress were not mounting labstore already
* 10:00 dcaro: removing labstore mounts on ingress-4
* 09:58 dcaro: removing labstore mounts on ingress-3
* 09:57 dcaro: removing labstore mounts on acme-chief
* 09:56 dcaro: removing labstore mounts on control-3
* 09:54 dcaro: removing labstore mounts on control-2

=== 2022-10-03 ===
* 14:05 Rook: remove labstore references #206  [[phab:T317881|T317881]] {{Gerrit|f891319851f2ad6e52a4e3000ea3a2341e262e45}}

=== 2022-09-27 ===
* 07:22 Rook: Upgrade jupyterhub chart #212  [[phab:T311266|T311266]] {{Gerrit|8de422c43525647cf839bbc3e82930626864fa73}}

=== 2022-09-26 ===
* 13:56 taavi: restart the 6 singleuser pods that don't have the new dumps mount points attached yet [[phab:T317144|T317144]]
* 12:42 Rook: Upgrade julia to 1.8.1 #210  {{Gerrit|51307a7d45ed651b4d27763559f3245d72d1aa5c}} [[phab:T318276|T318276]]
* 11:15 Rook: bump pywikibot version #211  [[phab:T318519|T318519]]   {{Gerrit|b540c81d134f1e24c9053d76bd3526186fa15bf6}}
* 08:42 Rook: Remove unused file #209  [[phab:T318277|T318277]] {{Gerrit|9f5034e4c72e5acbad1776bde19042eb9e9f36af}}

=== 2022-09-20 ===
* 18:43 Rook: Bump oauthlib from 3.2.0 to 3.2.1 in /images/minesweeper #207  {{Gerrit|dcfcfaa95da0ee89c50d229d369fcebe0a50232c}}

=== 2022-08-31 ===
* 14:10 Rook: updating openrefine install method to unblock nbclassic [[phab:T312251|T312251]] {{Gerrit|7aa340ca2643e90f9aa3534c6557c142847d1560}}
* 11:44 Rook: jupyterhub version bump [[phab:T316309|T316309]] {{Gerrit|5308be406ca5cdb0bfa823c7040e92fa19c191c9}}

=== 2022-08-29 ===
* 18:48 Rook: jupyterlab version bump {{Gerrit|0b2532f0590fc5c1ae5b5f5a7148bdf9f4b1f92c}} [[phab:T308042|T308042]]

=== 2022-08-25 ===
* 15:05 Rook: OpenRefine version bump [[phab:T316218|T316218]]  {{Gerrit|55271d2fe80a04f7b2b17426a882f81b4ca8a413}}
* 09:54 Rook: resource requests for renderer {{Gerrit|3598be38693668e1733c34fc6df13318db74f17e}} [[phab:T315682|T315682]]

=== 2022-08-24 ===
* 13:44 Rook: Upgrade ingress-nginx to 1.3.0 {{Gerrit|8e6b577fbc8a48167919dd20365967d68eb60a7d}}

=== 2022-08-23 ===
* 11:40 Rook: setup requests for nbserve [[phab:T315670|T315670]] {{Gerrit|e558ee7f619a590838e106c9866e2f8ebae33e58}}

=== 2022-08-22 ===
* 12:40 Rook: Upgrade pywikibot 7.5.0 -&gt; 7.6.0 #193  {{Gerrit|456d3f2fe0}} [[phab:T315745|T315745]]
* 11:44 Rook: updating key method #190 {{Gerrit|5631062a8a}} [[phab:T312096|T312096]]

=== 2022-08-04 ===
* 12:26 Rook: resource requests for quota {{Gerrit|cc8556d3ae422b99bb60c26d6ac0ce6f1e285b07}} [[phab:T314432|T314432]]

=== 2022-08-02 ===
* 12:22 Rook: deleting worker-1 from cluster and shutting down [[phab:T313287|T313287]]

=== 2022-07-25 ===
* 20:25 Rook: [[phab:T313728|T313728]] #188 revert nbclassic to restore openrefine
* 11:18 Rook: {{Gerrit|9336306f7be6791687f411dcd1db6735b670f2c9}} update pywikibot

=== 2022-07-22 ===
* 15:43 Rook: updating to give hub pod a cpu request so that it cannot be resource starved

=== 2022-07-19 ===
* 08:12 taavi: drain paws-k8s-worker-1 [[phab:T313287|T313287]]

=== 2022-07-14 ===
* 14:21 Rook: {{Gerrit|c6ef2b22b1ab3671d3f463fb2a0fa67aec265caa}} update nbclassic [[phab:T312251|T312251]]

=== 2022-07-13 ===
* 18:25 Rook: {{Gerrit|002d1a8261cfb701a64959d7e53c876d9a63393e}} update dockerfile to give julia and openrefine sections
* 11:59 Rook: {{Gerrit|cd8ec3987350e71e3798bfbcea0e65a9d5fa6b52}} node from its own repo #182

=== 2022-07-12 ===
* 13:12 Rook: {{Gerrit|2e11fb076f6b3cda1b538f58d022bf23e417361a}} libssl from repo #181 [[phab:T312236|T312236]]

=== 2022-07-11 ===
* 14:09 Rook: refactor: create R install section #180 {{Gerrit|bac49ef0f2a2daeb0d5329c4f64d6c4591c2daa0}} [[phab:T312570|T312570]]
* 11:15 Rook: {{Gerrit|08f1e33a8f408275f257c0911c1fb4f991e70f1e}} reorder minimal install #179 [[phab:T312593|T312593]]

=== 2022-07-07 ===
* 20:56 Rook: SPARQL to own section of build {{Gerrit|d3827e3fc6a137e54c997b4b935d9583b9a0e9b6}} [[phab:T312569|T312569]]
* 14:26 Rook: update pwb version {{Gerrit|6fcb349aef861ee229ce856672720586223dd9d4}} [[phab:T311958|T311958]]
* 11:17 Rook: unblock singleuser container build [[phab:T312088|T312088]] {{Gerrit|977c8d4dc6f38be96f29914ee0cafbb7d837b06d}}

=== 2022-06-27 ===
* 16:42 Rook: Make renderer accessable in dev env #172 {{Gerrit|9f2f171ff2a519fa11a784e8236eaf337bb13d08}}  [[phab:T308986|T308986]]

=== 2022-06-23 ===
* 13:20 Rook: remove deployhook token #171 [[phab:T310382|T310382]] {{Gerrit|74180f43cf6487f53b4812106ab8471aba2f8528}}

=== 2022-06-21 ===
* 12:36 Rook: [[phab:T302164|T302164]] #173 upgrading single user container {{Gerrit|b687ca6aedc745c63b2659124dca0dab01d38173}}

=== 2022-06-15 ===
* 18:03 Rook: removing unnecessary yaml #167 {{Gerrit|f4733023a1b63e48d12f99da5344af5513aea245}}
* 17:04 Rook: Move renderer to ubuntu container #168 {{Gerrit|078808991a683f44564a1dd8b3d13e9c227617c7}}

=== 2022-06-14 ===
* 13:11 Rook: Removing leftover db-proxy bits #170 {{Gerrit|8795bf7ded5eb2406a7dce70d8bd4450f6b2511d}}

=== 2022-06-07 ===
* 13:01 Rook: remove db-proxy #166 [[phab:T309794|T309794]]  {{Gerrit|643e112be91c355b8769caeab9e814dea9172b7d}}

=== 2022-06-06 ===
* 13:54 Rook: [[phab:T308975|T308975]] Move away from git to pip for nbconvert #162 {{Gerrit|54f93d02f62ec98d789a2e64a1288dff763833db}}
* 12:21 Rook: [[phab:T308926|T308926]] Move nbserve to upstream container #165 {{Gerrit|1767870b6724e863e705e86bf85c9e0bb90bafdf}}

=== 2022-06-02 ===
* 17:02 Rook: scaling db-proxy to zero [[phab:T309794|T309794]]

=== 2022-05-22 ===
* 12:59 Rook: deploying [[phab:T308862|T308862]]: Fix version of nbvconvert #158 {{Gerrit|47a7db6b320e70a4183b0a1891096ec5b00129fe}}

=== 2022-05-18 ===
* 12:55 Rook: bump jupyterhub version #149 [[phab:T308568|T308568]] {{Gerrit|41f03a544041318f1fad479b32ae46ac9e816a55}}
* 11:10 Rook: updating ingress from v1beta1 #134 [[phab:T294342|T294342]] {{Gerrit|5107562ccaf1160ba09a62ac7e2de9cd5e90f79f}}

=== 2022-05-16 ===
* 09:36 dcaro: restarted reload-acme-chief-backend.service to ensure certs are refreshed

=== 2022-05-14 ===
* 16:16 andrewbogott: restarting acme-chief.service on paws-acme-chief-01 for [[phab:T308383|T308383]]

=== 2022-05-11 ===
* 13:44 Rook: update pwb version and pin jupyterlab version {{Gerrit|ef3e38c671e20fad6b8297dfb20a14a913a8dda5}}

=== 2022-05-10 ===
* 13:56 Rook: upgrade pywikibot on container start {{Gerrit|437f46a1caf56cf860f85211af4567735410a621}}

=== 2022-04-27 ===
* 17:02 Rook: pywikibot version bump {{Gerrit|3c42a6243161ce6f6d8ff39829a58fd576d161b5}}

=== 2022-04-18 ===
* 17:29 Rook: updating links to phab with prefilled ticket links {{Gerrit|aef7c671a69a66a9872a48a24169f1f6bf2ffc4f}}
* 12:30 Rook: update pywikibot {{Gerrit|6db74b6c866021686a49aa8a2c4eb2d5da3bdddb}}

=== 2022-04-16 ===
* 18:36 wm-bot: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics ({{Gerrit|2c485e9}}) - cookbook ran by taavi@runko

=== 2022-04-04 ===
* 12:31 taavi: moving all VMs from paws-puppetmaster-01 -&gt; paws-puppetmaster-2

=== 2022-03-29 ===
* 10:36 Rook: upgrading pywikibot {{Gerrit|702f21dfe79e558d504c34474b295f1b8a5f75bb}}

=== 2022-03-21 ===
* 11:11 Rook: deploying jupyterlab {{Gerrit|cd6ee196a66c8a7d48504166fd776973054535a4}}

=== 2022-03-10 ===
* 12:23 Rook: updating banner to note ui will update soon {{Gerrit|462ab184e59a7f791bbc8790e823146f6e482920}}

=== 2022-03-08 ===
* 13:26 Rook: upgrading open refine {{Gerrit|c116d64f07dce58e884be64d0cdee70c79739dd4}}

=== 2022-03-07 ===
* 11:12 Rook: deploying paws realtime collaboration {{Gerrit|246e2afa97060dbd010b5a36ac88589f972d741f}}

=== 2022-03-02 ===
* 14:20 Rook: deploying fixed version of jupyter-rsession-proxy {{Gerrit|abe89f6a70d75492c1ad1baa2bc955204bce3c2e}}

=== 2022-03-01 ===
* 13:38 Rook: deploying pyaudio fix {{Gerrit|978fb648dbd1d1f351bba74b6e2aff4023137087}}

=== 2022-02-23 ===
* 13:13 Rook: deploying {{Gerrit|e6eedbc58bd6f1f074912f4faf8075275ae13819}} cleanup

=== 2022-02-15 ===
* 18:20 chicocvenancio: added psp for minesweeper
* 16:04 mdipietro: updating pywikibot {{Gerrit|2fc27c99db35515121f5b80c9f3df5e648e56b58}}
* 14:21 chicocvenancio: Deploying minesweeper

=== 2022-01-25 ===
* 14:30 mdipietro: deployed {{Gerrit|93d33c44a876fda8928719353ac0d5cf75651fd7}} PR122

=== 2021-12-28 ===
* 14:14 taavi: deploying https://github.com/toolforge/paws/pull/121

=== 2021-12-20 ===
* 18:20 majavah: deploying calico v3.21.0 ([[phab:T292698|T292698]])

=== 2021-12-16 ===
* 12:55 mdipietro: deploying {{Gerrit|dcac479b8c68631977441be5aaff8f6335659e1a}} [[phab:T294431|T294431]]

=== 2021-12-02 ===
* 19:12 chicocvenancio: deploy PR 111 [[phab:T295257|T295257]]
* 12:52 mdipietro: upgrading pywikibot {{Gerrit|0f5d28d23f49f9ba4008d48f9ff8323dd48c702b}}

=== 2021-12-01 ===
* 11:36 mdipietro: deploying lsof pr-76 {{Gerrit|a378845f87ff57427b44b8a8f4b616a0a74cfbf4}}

=== 2021-11-29 ===
* 13:02 chicocvenancio: deploy PR 113 [[phab:T295761|T295761]]
* 12:29 chicocvenancio: deploy PR 112 [[phab:T295761|T295761]]

=== 2021-11-25 ===
* 21:37 chicocvenancio: rollback singleuser to PR #96  [[phab:T295257|T295257]]
* 21:15 chicocvenancio: deploy PR #110 changing singleuser to bump openrefine version [[phab:T295257|T295257]]

=== 2021-11-23 ===
* 14:19 mdipietro: increased cull timeout with deploy of {{Gerrit|3e5726489d370db3a11f166d90190f6271e5a743}}

=== 2021-11-22 ===
* 12:57 mdipietro: added julia to paws with {{Gerrit|7b58fb09292a27f939f93ffba6ff00f8028302f2}}
* 11:04 mdipietro: added julia to paws with {{Gerrit|12bfdad0b4d7c60737b79914b6098029235ce9ef}}

=== 2021-11-11 ===
* 08:35 majavah: disabling pod preset controller in preparation for [[phab:T291913|T291913]]

=== 2021-11-09 ===
* 16:24 mdipietro: deployed PR97 ({{Gerrit|85c085f5309d45c598c3089fd64fa4810c769c06}}) Update Pywikibot to 6.6.2

=== 2021-11-03 ===
* 12:41 mdipietro: deployed https://github.com/toolforge/paws/pull/92 ({{Gerrit|4961101972be6f27c1c96e327a82457346965f32}}) [[phab:T150098|T150098]]

=== 2021-11-01 ===
* 12:31 majavah: upgrade ingress-nginx [[phab:T292771|T292771]]

=== 2021-10-28 ===
* 14:35 chicocvenancio: set team toolforge/wmcsadmins as maintainers for github repo

=== 2021-10-26 ===
* 15:06 chicocvenancio: delete orphan pods for 2 users

=== 2021-10-22 ===
* 16:38 majavah: deploy https://github.com/toolforge/paws/pull/91

=== 2021-10-21 ===
* 12:58 mdipietro: upgraded to {{Gerrit|923250f7cd0b522259abdad450fbd2dfb16357bf}} which was really not an upgrade as the diff gave nothing. Though now it is clear what is deployed.

=== 2021-09-07 ===
* 22:14 bstorm: upgraded k8s to 1.19.13 [[phab:T287399|T287399]]

=== 2021-08-18 ===
* 19:09 bstorm: redeployed hub with trove database backend instead of toolsdb

=== 2021-07-29 ===
* 14:09 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]

=== 2021-07-25 ===
* 16:09 majavah: deleting ingress pod running on worker-6 to get it to re-appear in ingress-4

=== 2021-07-21 ===
* 19:53 bstorm: deployed new maintain-kubeusers [[phab:T285011|T285011]]
* 19:53 bstorm: deployed new rbac for maintain-kubeusers changes [[phab:T285011|T285011]]
* 16:59 majavah: deploying calico v3.18.4 [[phab:T280342|T280342]]
* 15:52 majavah: add my key to passwords::root::extra_keys
* 15:00 majavah: starting kubernetes upgrades [[phab:T280302|T280302]]

=== 2021-07-14 ===
* 10:38 majavah: correction: undeploy old ingress [[phab:T264221|T264221]]
* 10:35 majavah: undeploy old ingress [[phab:T266050|T266050]]

=== 2021-07-13 ===
* 07:51 majavah: renewing tools-prometheus certificates

=== 2021-07-12 ===
* 13:18 majavah: ingress upgrade completed
* 13:05 majavah: moving user traffic to updated ingress-nginx [[phab:T264221|T264221]]

=== 2021-07-01 ===
* 12:04 majavah: deploy ingress-nginx 0.46 via the helm chart to paws [[phab:T264221|T264221]]

=== 2021-06-30 ===
* 20:05 bstorm: tried force delete on the ingress-nginx-gen2 namespace, which doesn't appear to be working either until metrics-server is fixed [[phab:T285905|T285905]]
* 20:00 bstorm: renewed k8s metrics-server certs and the deployment
* 18:04 majavah: renew kubernetes metrics-server certificate
* 17:26 majavah: creating paws-k8s-ingress-[3-4] and joining them to the k8s cluster [[phab:T264221|T264221]]
* 17:16 bstorm: temporarily increased quota to 60 cores to enable [[phab:T264221|T264221]]

=== 2021-06-03 ===
* 20:43 chicocvenancio: tagged new singleuser image, fixes [[phab:T283969|T283969]]

=== 2021-05-27 ===
* 21:53 bstorm: added paws-k8s-control-2.paws.eqiad.wmflabs back to the list of control nodes at the proxy
* 21:50 bstorm: renewed the certs for paws-k8s-control-2
* 20:37 bstorm: removed paws-k8s-control-2.paws.eqiad.wmflabs from the proxy because it is somewhat broken (certs expired)
* 19:41 bstorm: forced removal of openrefine in paws for now and deleted all current user server pods to force use of the new image

=== 2021-05-23 ===
* 01:45 bstorm: manually pulling updated singleuser images because I hate waiting to test things
* 01:36 bstorm: update latest tag for singleuser image to {{Gerrit|b894471}} for https://github.com/toolforge/paws/pull/69
* 00:56 bstorm: deploying changes for https://github.com/toolforge/paws/pull/67

=== 2021-05-21 ===
* 00:06 bstorm: creating trove mysql instance pawsdb-1 [[phab:T267683|T267683]]

=== 2021-05-12 ===
* 19:33 bstorm: added taavi to paws.admin

=== 2021-05-11 ===
* 09:17 Majavah: set `profile::wmcs::kubeadm::docker_vol: false` on ingress nodes [[phab:T282087|T282087]]
* 09:15 arturo: added user `taavi` (Majavah) as projectadmin

=== 2021-04-20 ===
* 23:18 bstorm: deploying updated paws-hub image for https://github.com/toolforge/paws/pull/63

=== 2021-04-02 ===
* 21:50 bstorm: deploying latest PRs to add a note on the wikireplicas changes

=== 2020-12-21 ===
* 20:27 bstorm: applied tuning for timeouts and elections on the k8s etcd pods of 300 for heartbeat and 3000 for elections [[phab:T267966|T267966]]

=== 2020-12-17 ===
* 02:22 bstorm: Set PAWS hub back to using mariadb [[phab:T266587|T266587]]

=== 2020-12-16 ===
* 18:21 chicocvenancio: move paws to sqlite while toolsdb is down.

=== 2020-12-10 ===
* 17:00 arturo: fixing /etc/kubernetes/kublet.conf and restarting kubelet in paws-k8s-control-1 ([[phab:T269865|T269865]])

=== 2020-12-05 ===
* 00:42 bd808: `kubectl delete po renderer-794886b9cd-9nc6c -n prod` after seeing lots of listen queue full errors in the pod logs.

=== 2020-11-30 ===
* 18:22 bstorm: 1.17 upgrade for kubernetes complete [[phab:T268669|T268669]]
* 17:25 bstorm: upgrading the worker nodes (this will likely kill services briefly when some pods are rescheduled) [[phab:T268669|T268669]]
* 17:14 bstorm: updated the calico-kube-controllers deployment to use our internal registry to deal with docker-hub rate-limiting [[phab:T268669|T268669]] [[phab:T269016|T269016]]
* 17:09 chicocvenancio: delete orphaned jupyter server pod `kubectl -n prod delete pod jupyter--45volutionoftheuniverse`. Respective server not running in jupyter admin UI.
* 16:31 bstorm: upgrading pods on paws-k8s-control-3 [[phab:T268669|T268669]]
* 16:17 bstorm: starting upgrade on paws-k8s-control-2 [[phab:T268669|T268669]] (first kubectl drain paws-k8s-control-2 --ignore-daemonsets)
* 15:53 bstorm: proceeding with upgrade to 1.17 on paws-k8s-control-1 [[phab:T268669|T268669]]
* 15:49 bstorm: draining paws-k8s-control-1 for upgrade [[phab:T268669|T268669]]
* 12:49 arturo: disable puppet in all k8s nodes to prepare for the upgrade ([[phab:T268669|T268669]])
* 12:49 arturo: set hiera `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-17'` at project level ([[phab:T268669|T268669]])

=== 2020-11-16 ===
* 22:13 bstorm: deploying new paws changes for multiinstance readiness

=== 2020-11-10 ===
* 20:16 chicocvenancio: restart hub to apply move to sqlite.  [[phab:T267667|T267667]]
* 16:41 arturo: set paws in sqlite mode because [[phab:T266587|T266587]] (kubectl --namespace prod edit configmap hub-config)

=== 2020-10-15 ===
* 19:12 andrewbogott: uncordoned paws-k8s-worker-1 and -2
* 18:48 andrewbogott: draining paws-k8s-worker-2 for move to ceph
* 18:36 andrewbogott: draining paws-k8s-worker-1 for move to ceph

=== 2020-09-29 ===
* 10:59 arturo: last 2 commands should help puppet agent in the paws project, previously it had issues fetching acme-chief certs because an API update
* 10:58 arturo: aborrero@paws-acme-chief-01:~$ sudo systemctl restart uwsgi-acme-chief.service
* 10:56 arturo: aborrero@paws-acme-chief-01:~$ sudo systemctl restart acme-chief.service

=== 2020-08-14 ===
* 17:09 bstorm: backing up the old proxy config to NFS and deleting paws-proxy-02 [[phab:T211096|T211096]]

=== 2020-08-07 ===
* 22:30 bstorm: removing downtime for paws and front page monitor [[phab:T211096|T211096]]
* 18:01 bstorm: shutting down paws-proxy-02 [[phab:T211096|T211096]]
* 17:05 bstorm: running the final rsync to the new cluster's nfs [[phab:T211096|T211096]]
* 16:08 bstorm: changing paws.wmflabs.org to point at the new cluster ip 185.15.56.57 [[phab:T211096|T211096]]
* 16:02 bstorm: LAST MESSAGE WRONG: switching NEW cluster to toolsdb [[phab:T211096|T211096]]
* 16:02 bstorm: switching old cluster to toolsdb [[phab:T211096|T211096]]
* 15:58 bstorm: switching old cluster to sqlite [[phab:T211096|T211096]]
* 15:53 bstorm: downtiming alerts in case they need changes (seems likely) [[phab:T211096|T211096]]

=== 2020-07-30 ===
* 20:40 bstorm: upgrading the singleuser image to test shuffling around some of the pip installs
* 16:38 bstorm: removing the *.paws.wmflabs.org SNI name because it won't be used and it might trigger a re-issue of certs [[phab:T255249|T255249]]
* 15:39 bstorm: upgrading acme-chief to 0.27-1

=== 2020-07-29 ===
* 18:03 bstorm: powering on paws-k8s-haproxy-1 because that worked fine
* 18:00 bstorm: powering off paws-k8s-haproxy-1 to test failover

=== 2020-07-24 ===
* 17:25 bstorm: to force repulling of every image everywhere, uninstalling paws in the new cluster and reinstalling it [[phab:T258812|T258812]]
* 09:39 arturo: dropped the DNS wildcard record `*.paws.wmcloud.org IN A 185.15.56.57` and created concrete CNAME records for the FQDNs we actually use ([[phab:T211096|T211096]])

=== 2020-07-23 ===
* 22:51 bstorm: deploying via the default 'latest' tag in the new cluster [[phab:T211096|T211096]]
* 22:48 bstorm: tagged the newbuild tags with &quot;latest&quot; to set sane defaults for all images in the helm chart [[phab:T211096|T211096]]
* 21:14 bstorm: pushing quay.io/wikimedia-paws-prod/nbserve:newbuild to main repo [[phab:T211096|T211096]]
* 21:11 bstorm: pushing quay.io/wikimedia-paws-prod/deploy-hook:newbuild to main repo [[phab:T211096|T211096]]
* 21:09 bstorm: pushing quay.io/wikimedia-paws-prod/singleuser:newbuild to the main repo [[phab:T211096|T211096]]
* 21:08 bstorm: pushing quay.io/wikimedia-paws-prod/paws-hub:newbuild to the main repo [[phab:T211096|T211096]]
* 21:06 bstorm: pushing dbproxy docker image for new cluster into main quay.io repo [[phab:T211096|T211096]]

=== 2020-07-22 ===
* 23:32 bstorm: setting the default NFS version to 4.2 while excepting the two stretch servers [[phab:T257945|T257945]]

=== 2020-07-21 ===
* 15:13 chicocvenancio: merge pr #50 to fix [[phab:T258142|T258142]]

=== 2020-07-06 ===
* 21:41 bstorm: deployed ingress to redirect paws.wmcloud.org to the wikitech doc page [[phab:T195217|T195217]]

=== 2020-06-30 ===
* 23:00 bstorm: added paws-public.wmflabs.org to the alt-names for acme-chief, which broke it until we hand off the zone to the paws project &lt;sorry!&gt; [[phab:T195217|T195217]] [[phab:T255997|T255997]]

=== 2020-06-26 ===
* 21:57 bstorm: applied the metrics manifests to kubernetes to enable metrics-server, cadvisor, etc. [[phab:T256361|T256361]]

=== 2020-06-25 ===
* 22:52 bstorm: created paws-k8s-worker-5/6/7 as x-large nodes to bring the cluster up to roughly the same capacity as the existing one using soft anti-affinity [[phab:T211096|T211096]] [[phab:T253267|T253267]]
* 22:43 bstorm: bumped quota up to 24 instances, 128 GB RAM and 56 cores [[phab:T211096|T211096]]
* 16:39 bstorm: deleted the deployhook from the in-progress new cluster for now just in case [[phab:T211096|T211096]]
* 15:44 bstorm: deployed a proof-of-concept paws-public setup in the new cluster [[phab:T255997|T255997]]

=== 2020-06-24 ===
* 23:18 bstorm: added A record for *.paws.wmcloud.org to public and hub to use [[phab:T211096|T211096]] [[phab:T255997|T255997]] [[phab:T195217|T195217]]
* 21:45 bstorm: doing an initial rsync of the paws userhomes to the new project [[phab:T160113|T160113]]

=== 2020-06-19 ===
* 10:01 arturo: enabled `paws.wmflabs.org` and `*.paws.wmflabs.org` as valid ingress domains (acme-chief TLS cert, haproxy, etc) ([[phab:T195217|T195217]])

=== 2020-06-17 ===
* 21:51 bstorm_: upgraded chart in the new cluster to include resource limits [[phab:T251298|T251298]]
* 21:51 bstorm_: upgraded chart in the new cluster to include resource limits

=== 2020-06-16 ===
* 15:48 arturo: change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 ([[phab:T195217|T195217]])
* 15:47 arturo: associate floating IP 185.15.56.57 with haproxy VIP port ([[phab:T295217|T295217]])
* 15:43 arturo: allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 1b40be58-7182-41aa-95ce-797f94f83d66` ([[phab:T295217|T295217]])
* 15:43 arturo: allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 9ccc43d9-1a8a-4287-afda-67e8bab27a9f` ([[phab:T295217|T295217]])
* 15:37 arturo: `aborrero@cloudcontrol1004:~ 1 $ sudo wmcs-openstack --os-project-id=paws port create --network 7425e328-560c-4f00-8e99-{{Gerrit|706f3fb90bb4}} paws-haproxy-vip` ([[phab:T295217|T295217]])
* 15:23 arturo: live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for [[phab:T195217|T195217]]

=== 2020-06-15 ===
* 15:59 arturo: created DNS record `deploy-hook.paws.wmcloud.org IN CNAME paws.wmcloud.org` ([[phab:T195217|T195217]])
* 12:28 arturo: manually created an Ingress object to test routing to the hub ([[phab:T195217|T195217]])
* 12:20 arturo: created DNS record `paws.wmcloud.org IN A 185.15.56.57` ([[phab:T195217|T195217]])
* 12:19 arturo: associate floating IP 185.15.56.57 with VM paws-k8s-haproxy-1 ([[phab:T195217|T195217]])
* 12:18 arturo: release floating IP not in use: 185.15.56.42
* 12:18 arturo: release floating IP not in use: 185.15.56.43
* 11:45 arturo: reset wikitech user password for the service account `paws-dns-manager` to what is in labs/private.git/hieradata/common.yaml `profile::acme_chief::cloud::designate_sync_password` ([[phab:T195217|T195217]])

=== 2020-06-12 ===
* 18:49 bstorm_: deployed a test of paws chart in the new cluster [[phab:T211096|T211096]]
* 13:23 arturo: assigned the DNS zone `paws.wmcloud.org` ([[phab:T195217|T195217]])
* 13:13 arturo: live-hacking session in the puppetmaster ended
* 13:05 arturo: live-hacking puppet tree in paws-puppetmaster-01 for [[phab:T195217|T195217]]
* 11:55 arturo: `aborrero@cloudcontrol1004:~ $ sudo wmcs-openstack role add --user paws-dns-manager --project paws observer` ([[phab:T255252|T255252]])
* 11:55 arturo: `aborrero@cloudcontrol1004:~ $ sudo wmcs-openstack role add --user paws-dns-manager --project paws designateadmin` ([[phab:T255252|T255252]])
* 11:51 arturo: created service account `paws-dns-manager` in wikitech ([[phab:T255252|T255252]])
* 11:31 arturo: introduced acme-chief private data into labs/private in paws-puppetmaster-01 ([[phab:T255252|T255252]])
* 11:02 arturo: created puppet prefix 'paws-acme-chief' ([[phab:T255252|T255252]])
* 11:01 arturo: created VM paws-acme-chief-01 ([[phab:T255252|T255252]])

=== 2020-06-11 ===
* 11:11 arturo: deployed nginx-ingress for some early testing (not definitive) with code https://github.com/crookedstorm/paws/commit/bee62b3fd57f9804aa27e7b8b41fde50bd93df94 ([[phab:T195217|T195217]])
* 10:15 arturo: added role (just a label) for ingress nodes: `kubectl label node paws-k8s-ingress-1 kubernetes.io/role=ingress` ([[phab:T195217|T195217]])

=== 2020-06-04 ===
* 14:16 arturo: added node taints to ingress nodes: `kubectl taint nodes paws-k8s-ingress-1 ingress=true:NoSchedule` ([[phab:T195217|T195217]])
* 12:18 arturo: bootstrapped paws-k8s-ingress nodes, added them to the k8s cluster ([[phab:T195217|T195217]])
* 12:04 arturo: created `paws-k8s-ingress` puppet prefix and add the `role::wmcs::paws::k8s::worker` role ([[phab:T195217|T195217]])
* 12:02 arturo: created 2 medium VM instances: paws-k8s-ingress-1 and paws-k8s-ingress-2 with haproxy anti-affinity ([[phab:T195217|T195217]])

=== 2020-05-26 ===
* 22:34 bstorm_: restored the deployment for maintain-kubeusers so anyone added to the paws.admin group will have admin on the cluster now that the bug is fixed [[phab:T211096|T211096]] [[phab:T246059|T246059]]
* 22:05 bstorm_: temporarily deleted the deployment for maintain-kubeusers pending patch to fix context creation for new admin accounts [[phab:T211096|T211096]] [[phab:T246059|T246059]]
* 22:04 bstorm_: created paws-focused PodSecurityPolicies and the prod namespace in the new cluster [[phab:T211096|T211096]]
* 22:03 bstorm_: created paws.admin group and kubernetes admin accounts on the new k8s cluster [[phab:T211096|T211096]] [[phab:T246059|T246059]]
* 18:29 bstorm_: bootstrapped the new control plane nodes [[phab:T211096|T211096]]
* 15:27 bstorm_: updated profile::wmcs::kubeadm::kubernetes_version to 1.16.10 for cluster init [[phab:T211096|T211096]]

=== 2020-05-21 ===
* 23:04 bstorm_: added profile::wmcs::kubeadm::k8s::encryption_key and profile::wmcs::kubeadm::k8s::node_token to labs/private [[phab:T211096|T211096]]
* 14:53 bstorm_: adding the hiera values to horizon for bootstrapping k8s [[phab:T211096|T211096]]
* 14:39 arturo: point record `k8s.svc.paws.eqiad1.wikimedia.cloud` to `172.16.1.186` (which is paws-k8s-control-1, for the initial bootstrap) ([[phab:T211096|T211096]])
* 12:48 arturo: created record `k8s.svc.paws.eqiad1.wikimedia.cloud` pointing to `172.16.0.191` (which is paws-k8s-haproxy-1) ([[phab:T211096|T211096]])
* 12:34 arturo: created and transferred DNS zone `svc.paws.eqiad1.wikimedia.cloud` ([[phab:T211096|T211096]])

=== 2020-05-20 ===
* 22:35 bstorm_: created paws-k8s-worker-1/2/3/4 [[phab:T211096|T211096]]
* 22:12 bstorm_: created paws-k8s-haproxy-1/2 with antiaffinity group [[phab:T211096|T211096]]
* 21:36 bstorm_: created paws-k8s-control-1/2/3 with appropriate sec group and server group [[phab:T211096|T211096]]
* 18:59 bstorm_: created anti-affinity group &quot;controlplane&quot; [[phab:T211096|T211096]]
* 16:38 bstorm_: deleting the old shut-down VMs from the last effort to rebuild paws [[phab:T211096|T211096]]
* 16:36 bstorm_: cleaned up the old DNS entries for the external LBs that have been off for a year

=== 2020-03-20 ===
* 14:03 jeh: upgrade paws-puppetmaster-01 to v5 [[phab:T241719|T241719]]

=== 2020-02-14 ===
* 21:31 andrewbogott: restarting paws-puppetmaster-01 so its clients can connect

=== 2020-01-09 ===
* 18:06 bstorm_: rebooting tools-paws-master-01 [[phab:T242353|T242353]]
* 14:28 chicocvenancio: shutdown unused instances

=== 2019-12-13 ===
* 00:27 bstorm_: rebooting the paws master since it is in a bad state after the openstack maintenance as well.

=== 2019-11-01 ===
* 21:15 Krenair: Updated paws-apiserver.wmflabs.org A record list to remove 172.16.2.151 which is not allocated to any instance. The other two A records point to valid instances in the paws project.

=== 2019-10-23 ===
* 09:03 arturo: paws-master-01/03 and a couple of other servers are down because hypervisor is rebooting

=== 2019-10-14 ===
* 22:32 bd808: Removed project member &quot;Afrodric&quot;. Looks like someone added accidentally when trying to make aborrero as project member
* 22:31 bd808: Added Krenair as project member

=== 2019-05-18 ===
* 11:13 chicocvenancio: point paws-proxy-02 to tools-paws-worker-1006 on paws-deploy-hook hostname ([[phab:T218380|T218380]])

=== 2019-04-26 ===
* 15:27 chicocvenancio: paws-proxy-02 redirecting main page to https://www.mediawiki.org/wiki/PAWS [[phab:T221886|T221886]]

=== 2019-04-16 ===
* 17:15 chicocvenancio: move paws-proxy-02 reload nginx
* 17:07 chicocvenancio: move paws-proxy-02 to point to tools-paws-worker-1006 for upcoming master move

=== 2019-03-27 ===
* 23:46 chicocvenancio: moving paws host in `paws-proxy-02`  back to `tools-paws-master-01` [[phab:T219460|T219460]]
* 22:10 chicocvenancio: moving paws host in `paws-proxy-02` to `tools-paws-worker-1005` [[phab:T219460|T219460]]

=== 2019-03-25 ===
* 14:12 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project ([[phab:T211096|T211096]])
* 14:07 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project [[phab:T211096|T211096]]
* 13:54 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project ([[phab:T211096|T211096]])

=== 2019-03-15 ===
* 02:25 gtirloni: activated TLS termination using Let's Encrypt on paws-proxy-02
* 02:25 gtirloni: removed webproxies and created new A records pointing directly to paws-proxy-02

=== 2019-02-21 ===
* 09:22 gtirloni: upgraded and rebooted paws-proxy-02

=== 2019-02-20 ===
* 15:00 andrewbogott: deleting the long-shut-down paws-proxy-01

=== 2019-02-15 ===
* 01:28 bd808: Re-enabled PAWS vhost on paws-proxy-02

=== 2019-02-14 ===
* 22:25 gtirloni: downtimed PAWS in Icinga
* 22:16 gtirloni: Activated maintenance page on paws-proxy-02 nginx config

=== 2019-02-13 ===
* 08:32 arturo: switch paws-proxy-02 puppetmaster to labs-puppetmaster.wikimedia.org

=== 2019-01-24 ===
* 19:20 andrewbogott: shutting down paws-proxy-01
* 19:11 chicocvenancio: moved config, ready to receive  traffic on paws-proxy-02  [[phab:T214613|T214613]]
* 18:34 chicocvenancio: firing up paws-proxy-02 for [[phab:T214613|T214613]]

=== 2019-01-23 ===
* 20:46 chicocvenancio: moving paws_public  proxy_pass to https://172.16.6.39 in paws-proxy-01

=== 2018-10-25 ===
* 23:58 gtirloni: Started tools-paws-worker-1010 ([[phab:T208006|T208006]])

=== 2018-08-03 ===
* 20:19 andrewbogott: deleting paws-master-01 and paws-node-1002; unused

=== 2018-07-03 ===
* 22:49 bstorm_: added stricter image space reclaiming arguments to kubelet

=== 2018-06-20 ===
* 17:39 chicocvenancio: edited paws-proxy-01 to pass http_x_forwarded_proto as it receives [[phab:T197248|T197248]]

=== 2018-05-04 ===
* 02:48 chicocvenancio: killed 25 pods with more than one hour inactivity through admin interface

=== 2018-03-14 ===
* 21:49 chicocvenancio: updated k8s control plane, updating nodes to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-02-23 ===
* 18:33 chicocvenancio: redirected tools.wmflabs.org/paws to paws.wmflabs.org and deleted old k8s ReplicationControllers ([[phab:T188068|T188068]])

=== 2018-02-22 ===
* 22:11 chicocvenancio: ([[phab:T175202|T175202]]) culler is running and killing pods as designed!
* 21:13 chicocvenancio: jupyterhub updated to fix culler ([[phab:T175202|T175202]]) culler already ran without 404
* 17:43 chicocvenancio: manually ran culler inside hub container

=== 2018-02-21 ===
* 17:03 chicocvenancio: deleted query-killer k8s deployment [[phab:T187818|T187818]]

=== 2018-02-16 ===
* 20:18 chicocvenancio: changed userhomes group for [[phab:T185434|T185434]]  workarround

=== 2018-02-15 ===
* 01:10 chicocvenancio: changed group of all userhome folders to tools.paws

=== 2018-02-04 ===
* 12:21 chicocvenancio: changed group of all userhome folders to tools.paws

=== 2017-12-19 ===
* 22:11 bd808: Killed tiller pod that was in crashloopbackoff

=== 2017-09-28 ===
* 21:25 andrewbogott: server docker restart on paws-node-1002; disk is full and docker is holding open a lot of deleted files

=== 2017-03-20 ===
* 21:25 andrewbogott: migrating paws-base-01 to labvirt1013

=== 2016-05-10 ===
* 20:36 bd808: Visit https://foo-pawsbase.wmflabs.org/ to see list of wikis in wikifarm
* 20:32 bd808: Fixed hiera config so https://wikidata-pawsbase.wmflabs.org routes correctly
* 15:47 YuviPanda: added bd808 to project and made as admin

{{SAL|Project Name=paws}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>skvjfbsanrwxqpo2rn9w6abgs8xunh0</sha1>
    </revision>
    <revision>
      <id>2435492</id>
      <parentid>2435491</parentid>
      <timestamp>2026-07-11T13:01:25Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>dcaro: removed the files pending deletion as the systemd timer seems to be failing, that cleared some space</comment>
      <origin>2435492</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="56236" sha1="qq09tt4206esu9gx0wqsbpq9jbnuxu4" xml:space="preserve">=== 2026-07-11 ===
* 13:01 dcaro: removed the files pending deletion as the systemd timer seems to be failing, that cleared some space
* 12:55 dcaro: Out of space on nfs, looking

=== 2026-07-06 ===
* 16:42 andrewbogott: running &quot;find . -name ubuntu24.img -delete&quot; and &quot;find . -name ubuntu22.img -delete&quot; in /srv/paws/project/paws/userhomes

=== 2026-07-02 ===
* 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]])

=== 2026-06-29 ===
* 13:15 andrewbogott: rm -rf /srv/paws/files-to-remove/* on paws-nfs-2

=== 2026-06-23 ===
* 19:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2026-06-22 ===
* 15:43 andrewbogott: rm -rf /srv/paws/files-to-remove/* for [[phab:T429578|T429578]]

=== 2026-05-28 ===
* 17:25 andrewbogott: switched proxies to new paws release featuring pywikibot 1.30.0

=== 2026-05-07 ===
* 21:29 andrewbogott: rebooting all worker nodes to pick up modprobe changes

=== 2026-04-28 ===
* 09:46 volans: killed 22 zombies pods stuck in Terminating status since many days and with no running container/process

=== 2026-03-13 ===
* 20:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1)
* 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm paws-127c-uwce57bvcgrt-master-0 (cluster eqiad1)
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm paws-127c-uwce57bvcgrt-master-0 (cluster eqiad1)

=== 2026-03-09 ===
* 08:31 godog: archive outdated https://gitlab.wikimedia.org/repos/cloud/paws in favor of github repo

=== 2025-12-24 ===
* 17:21 andrewbogott: deleting everything in /srv/paws/files-to-remove [[phab:T396051|T396051]]

=== 2025-12-23 ===
* 10:56 dhinus: hard-reboot paws-127c-uwce57bvcgrt-node-1 (reporting NotReady in kubectl)

=== 2025-12-02 ===
* 20:50 andrewbogott: stop/starting paws-127b-rpchztfjt2jb-node-1 to fix mtu mismatch
* 20:00 andrewbogott: rebooting paws-127b-rpchztfjt2jb-master-0.paws to aid with host draining

=== 2025-11-19 ===
* 09:32 dcaro: reboot worker 2 and 4
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-2 (cluster eqiad1, project paws)
* 09:31 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-2 (cluster eqiad1, project paws)
* 09:31 dcaro: reboot worker 1
* 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)

=== 2025-10-08 ===
* 12:35 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]]

=== 2025-10-02 ===
* 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-4 (cluster eqiad1, project paws)
* 09:27 dcaro: reboot node-4 ([[phab:T406191|T406191]])

=== 2025-09-26 ===
* 09:17 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)

=== 2025-09-25 ===
* 16:56 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)
* 16:56 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws)
* 16:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-09-22 ===
* 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1, project paws)
* 08:54 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1, project paws)

=== 2025-09-18 ===
* 19:37 andrewbogott: replaced previous deployment with new 127c deployment, supporting ssh access to VMs for [[phab:T405023|T405023]]
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 18:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-09-15 ===
* 17:19 andrewbogott: hard rebooting paws worker nodes one by one
* 12:56 dcaro: restarting the k8s workers 1, 2 and 4
* 12:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127b-rpchztfjt2jb-node-4 (cluster eqiad1, project paws)
* 12:56 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127b-rpchztfjt2jb-node-4 (cluster eqiad1, project paws)
* 12:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127b-rpchztfjt2jb-node-2 (cluster eqiad1, project paws)
* 12:54 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127b-rpchztfjt2jb-node-2 (cluster eqiad1, project paws)
* 12:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127b-rpchztfjt2jb-node-1 (cluster eqiad1, project paws)
* 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm paws-127b-rpchztfjt2jb-node-1 (cluster eqiad1, project paws)

=== 2025-09-10 ===
* 08:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'volans' in role 'member'
* 08:16 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'volans' in role 'member'

=== 2025-09-09 ===
* 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 12:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0)
* 12:16 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot
* 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0)
* 12:15 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot
* 12:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2025-08-04 ===
* 14:23 andrewbogott: service restored
* 14:19 andrewbogott: restarting all paws k8s nodes
* 14:19 andrewbogott: extended paws-nfs volume ran e2fsck and resize2fs on paws-nfs-1 and rebooted
* 14:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 14:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
* 14:05 andrewbogott: stopping all nodes to clear the way for NFS maintenance
* 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]])
* 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]])

=== 2025-07-29 ===
* 20:19 andrewbogott: removing Alex Monk from project in response to his email request
* 09:00 taavi: deploy pywikibot 10.2.0 update [[phab:T394614|T394614]]

=== 2025-07-25 ===
* 14:26 andrewbogott: moving all home dirs with files named 'android' into /srv/paws/files-to-remove/

=== 2025-07-24 ===
* 12:40 andrewbogott: purging /srv/paws/files-to-remove

=== 2025-07-13 ===
* 03:31 andrewbogott: rebooted all paws worker nodes, which may or may not have fixed the service (it was down before and is back now!)

=== 2025-07-03 ===
* 12:53 andrewbogott: moved all notebooks with Android binaries to /srv/paws/files-to-remove/

=== 2025-06-30 ===
* 09:42 dcaro: deleted the collected files-to-remove due to 100% capacity on nfs

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2024-06-20 ===
* 11:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0)
* 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs

=== 2024-06-18 ===
* 20:46 andrewbogott: rebuilt magnum cluster using g4/ovs flavors

=== 2024-05-06 ===
* 17:15 Rook: upgrade pywikibot [[phab:T364188|T364188]]

=== 2024-04-29 ===
* 17:54 Rook: upgrade pywikibot [[phab:T363131|T363131]]
* 17:54 Rook: jupyterlab to 4.1.8 [[phab:T363596|T363596]]
* 16:50 Rook: k8s to 1.26 [[phab:T326985|T326985]]

=== 2024-03-25 ===
* 15:59 Rook: upgrade jupyter chart. New cluster [[phab:T360643|T360643]]
* 13:01 Rook: use upstream jupyter-rsession-proxy [[phab:T360800|T360800]]

=== 2024-03-15 ===
* 16:16 Rook: upgrade jupyterlab [[phab:T360193|T360193]]

=== 2024-03-09 ===
* 16:06 Rook: increase number of worker nodes [[phab:T359747|T359747]]

=== 2024-03-08 ===
* 12:33 Rook: upgrade pywikibot [[phab:T359616|T359616]]
* 11:49 Rook: upgrade jupyterlab version [[phab:T359588|T359588]]

=== 2024-03-07 ===
* 21:01 Rook: increase worker count to manage outreachy load [[phab:T359591|T359591]]

=== 2024-03-06 ===
* 12:07 Rook: increase capacity for outreachy [[phab:T359316|T359316]]

=== 2024-03-05 ===
* 13:25 Rook: remove jupyter-dash [[phab:T358621|T358621]]

=== 2024-03-04 ===
* 13:57 Rook: add wikibase-cli [[phab:T358649|T358649]]

=== 2024-02-26 ===
* 16:09 Rook: remove nbextension moving away from notebook interface [[phab:T312234|T312234]]

=== 2024-02-21 ===
* 16:07 Rook: increase prometheus retention [[phab:T357786|T357786]]

=== 2024-02-20 ===
* 16:17 Rook: upgrade jupyterlab [[phab:T357990|T357990]]

=== 2024-02-15 ===
* 18:33 Rook: update openresty [[phab:T357698|T357698]]
* 13:59 Rook: upgrade jupyterlab to 4.1.1 [[phab:T357027|T357027]]

=== 2024-02-12 ===
* 15:56 Rook: upgrade OpenRefine [[phab:T356448|T356448]]

=== 2024-02-09 ===
* 17:48 Rook: jupyterlab upgraded to 4.1.0 [[phab:T357027|T357027]]

=== 2024-02-01 ===
* 20:17 Rook: prometheus and kube-state-metrics internal to cluster [[phab:T355179|T355179]]

=== 2024-01-29 ===
* 12:46 Rook: update jupyerlab [[phab:T355890|T355890]]

=== 2024-01-26 ===
* 15:25 Rook: update to allow for s3 tofu state storage in codfw1dev [[phab:T355543|T355543]]

=== 2024-01-24 ===
* 12:56 Rook: Remove 123-11 cluster [[phab:T355785|T355785]]

=== 2024-01-18 ===
* 08:05 Rook: upgrade rstudio-server [[phab:T355288|T355288]]

=== 2024-01-12 ===
* 14:00 Rook: removed paws-123-10 cluster [[phab:T354946|T354946]]

=== 2023-12-08 ===
* 14:41 Rook: upgrade OpenRefine [[phab:T353021|T353021]]

=== 2023-12-06 ===
* 14:28 Rook: pywikibot to 8.6 [[phab:T352794|T352794]]

=== 2023-11-27 ===
* 12:33 Rook: jupyterlab to 4.0.9 [[phab:T351726|T351726]]

=== 2023-11-15 ===
* 19:46 Rook: move to opentofu [[phab:T351249|T351249]]

=== 2023-11-13 ===
* 18:17 Rook: remove old cluster [[phab:T350875|T350875]]
* 13:21 Rook: pwb version bump [[phab:T351015|T351015]]

=== 2023-11-09 ===
* 14:34 Rook: updated ingress-nginx [[phab:T347506|T347506]]

=== 2023-11-06 ===
* 17:10 Rook: pywikibot to 8.5.0 [[phab:T350552|T350552]]

=== 2023-11-03 ===
* 09:27 Rook: bump jupyterlab version [[phab:T350459|T350459]]

=== 2023-10-24 ===
* 13:03 Rook: removed old cluster [[phab:T349551|T349551]]

=== 2023-10-23 ===
* 18:35 Rook: deploy new cluster/jupyterhub chart [[phab:T349545|T349545]]

=== 2023-10-19 ===
* 15:37 Rook: Bump jupyterlab version [[phab:T349203|T349203]]
* 14:53 Rook: Bump urllib3 from 1.26.17 to 1.26.18
* 12:59 Rook: Bump urllib3 from 1.26.16 to 1.26.17

=== 2023-10-18 ===
* 12:31 Rook: Jupyterlab to 3.0.6 [[phab:T347108|T347108]]
* 11:01 Rook: update OpenRefine [[phab:T348464|T348464]]

=== 2023-10-09 ===
* 15:50 Rook: bump pwb version [[phab:T348372|T348372]]

=== 2023-09-29 ===
* 09:58 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:56 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 09:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-09-21 ===
* 14:24 Rook: bump pwb version [[phab:T346912|T346912]]

=== 2023-09-05 ===
* 12:19 Rook: Reduce memory request for single user container [[phab:T345467|T345467]]

=== 2023-09-01 ===
* 18:45 Rook: increased worker node count to as all memory was requested [[phab:T345462|T345462]]

=== 2023-08-31 ===
* 16:51 taavi: restart uwsgi-toolsdb-replica-cnf-web.service on paws-nfs-2, a required dependency was installed only after last restart which made the service fail
* 13:52 Rook: pwb to version 8.3.2 [[phab:T345192|T345192]]

=== 2023-08-30 ===
* 12:39 Rook: Upgrade notebook allowing i18n support [[phab:T121478|T121478]]

=== 2023-08-28 ===
* 13:37 Rook: upgrade node [[phab:T341716|T341716]]

=== 2023-08-22 ===
* 18:06 Rook: bump jupyterlab version [[phab:T344265|T344265]]

=== 2023-08-21 ===
* 20:03 Rook: pywikibot version bump [[phab:T344493|T344493]]

=== 2023-08-18 ===
* 14:07 Rook: unrar added

=== 2023-08-15 ===
* 14:08 Rook: [[phab:T343116|T343116]] update helm chart, jupyterlab, jupyterhub, notebook. Dropping sparql to allow for update.

=== 2023-07-28 ===
* 12:26 Rook: pwb version bump [[phab:T342852|T342852]]

=== 2023-07-17 ===
* 18:33 Rook: update OpenRefine version [[phab:T341985|T341985]]

=== 2023-07-13 ===
* 14:06 Rook: terraform fmt [[phab:T341784|T341784]]
* 12:47 Rook: move db to pvc [[phab:T341132|T341132]]

=== 2023-07-12 ===
* 15:22 Rook: paws deploying from tf in codfw1dev [[phab:T340657|T340657]]

=== 2023-07-10 ===
* 11:18 Rook: shut down systems associated with [[phab:T341457|T341457]] should only be redirect page.
* 11:00 Rook: removed paws-db-backup-1 [[phab:T340054|T340054]]

=== 2023-06-30 ===
* 18:51 Rook: https://wikitech.wikimedia.org/wiki/PAWS/Admin updated to describe blue green deployment method using terraform
* 18:48 Rook: terraform working in eqiad1 [[phab:T340836|T340836]]
* 18:18 Rook: bounced control node as it was getting disk pressure

=== 2023-06-29 ===
* 13:52 Rook: [[phab:T340698|T340698]] OpenRefine version bump

=== 2023-06-13 ===
* 18:16 Rook: Rolling jupyterlab back to last known working version [[phab:T338981|T338981]]

=== 2023-06-12 ===
* 16:02 Rook: updated jupyterlab [[phab:T338337|T338337]]

=== 2023-06-07 ===
* 10:30 Rook: upgrade jupyterlab [[phab:T324002|T324002]]

=== 2023-05-15 ===
* 12:00 Rook: implemented limits to storage. Cleared about half the storage used [[phab:T327936|T327936]]

=== 2023-05-03 ===
* 12:26 Rook: upgrade jupyterhub {{Gerrit|fb5c7a33b4f6040b20ae19f4d3c368d3e3c0df29}} [[phab:T335810|T335810]]

=== 2023-04-24 ===
* 13:28 Rook: pywikibot version bump {{Gerrit|546696214f1a5e50e297241d22042a7cee870e11}} [[phab:T335237|T335237]]
* 11:37 andrewbogott: rebooting paws-k8s-haproxy-2, paws-acme-chief-01, paws-k8s-haproxy-1 -- their load numbers are through the roof for no obvious reason

=== 2023-04-17 ===
* 11:13 Rook: robots.txt to nbserve {{Gerrit|1fbc7865aad1ea592d9a852dda3bb3386fc1f29c}}

=== 2023-04-13 ===
* 21:19 Rook: revert jupyterlab upgrade, causing notebooks to not load [[phab:T324002|T324002]]

=== 2023-04-12 ===
* 14:42 Rook: upgrade jupyterhub and jupyterlab {{Gerrit|c3450668212846ef1430941d17f9890274247a4d}} [[phab:T324002|T324002]]
* 10:51 Rook: remove focal from R {{Gerrit|7a88b67d990035eb912c5399f5ea3deeb4b75f37}} [[phab:T331363|T331363]]

=== 2023-04-10 ===
* 12:40 Rook: openrefine version bump {{Gerrit|771250537d42d16bed997c3bd1f7f500512e634b}} [[phab:T334145|T334145]]

=== 2023-04-03 ===
* 14:31 Rook: pywikibot version bump {{Gerrit|c385d673b9a9fedd29f7ef6e586f527a93ae2f50}} [[phab:T333482|T333482]]

=== 2023-03-27 ===
* 19:41 Rook: Upgrade Wikimedia Commons extension for OpenRefine on PAWS to version 0.1.1 {{Gerrit|4b10d3f0d272d7dfb88022eb123232ff9b2f202f}} [[phab:T332721|T332721]]
* 17:50 Rook: bump pwb version {{Gerrit|7c4951863ae5a356c5b978f41b5eb4b465a72138}} [[phab:T333067|T333067]]

=== 2023-03-20 ===
* 13:12 Rook: move paws to k8s 1.23 {{Gerrit|b1d0daeb20cfeb6981653d65e44cf6175eded539}} [[phab:T328489|T328489]]

=== 2023-03-16 ===
* 13:13 Rook: pywikibot version bump {{Gerrit|6de17dbcbeba8da9e79cfae637945f7da4afc851}}

=== 2023-03-15 ===
* 16:46 Rook: Install Wikimedia Commons extension for OpenRefine {{Gerrit|aaae2fd0fb198fcf37efaba3d910fd0306c6763b}} Install Wikimedia Commons extension for OpenRefine
* 16:08 Rook: openrefine version bump {{Gerrit|9047213e25cd0a5ecc342ceb62f1a962c303d261}} [[phab:T331747|T331747]]

=== 2023-03-14 ===
* 14:21 Rook: NFS cutover {{Gerrit|0ab68a892bf40c6237d8edf5d9c23f2cbc430da0}} [[phab:T331056|T331056]]

=== 2023-03-13 ===
* 10:40 Rook: Restructure paws away from special networking (Change paws domain name) {{Gerrit|df16f355de3856c9ef7ef72ea4ae86dc9080723f}} [[phab:T328842|T328842]]

=== 2023-02-20 ===
* 21:08 Rook: upgrade openrefine to 3.7.0 {{Gerrit|adc41fc9ce208197589adae938bbc8374ae80c5b}} [[phab:T329835|T329835]]

=== 2023-02-18 ===
* 20:48 Rook: revert to installing pywikibot from git {{Gerrit|a22cd26b16dc256f505e70d30c3d10e038da747d}} [[phab:T326512|T326512]]

=== 2023-02-16 ===
* 12:58 Rook: moving to new cluster. Old one was restarting hub and couldn't find all of its nodes

=== 2023-02-14 ===
* 13:27 Rook: Bump oauthlib {{Gerrit|97f241bacff7af60cebfffa0d8eb945c7545577d}}
* 08:15 taavi: empty profile::wmcs::paws::control_nodes hiera key to bring PAWS back up ([[phab:T329581|T329581]]), it contained the hostnames of the old kubeadm backed cluster which should be cleaned up properly in [[phab:T327674|T327674]]

=== 2023-02-13 ===
* 12:20 Rook: pywikibot to version 8 {{Gerrit|16f07db23b6e380a15297605cbbb532f312a3246}} [[phab:T326512|T326512]]

=== 2023-01-31 ===
* 12:24 Rook: cluster upgraded to k8s 1.22 {{Gerrit|360b1238e85a7b3613db88a4919e87801ae69321}} [[phab:T327264|T327264]]

=== 2023-01-30 ===
* 13:51 Rook: Set 1 to = 1 for nfs mounts and variables {{Gerrit|09b036eb9dab09df39e9064cad4d18bc0db1b763}} [[phab:T326675|T326675]]
* 11:53 Rook: updated ingress-nginx to allow larger file (more than 800K) uploads [[phab:T328168|T328168]]

=== 2023-01-25 ===
* 15:00 Rook: deploy nginx config as configmap {{Gerrit|1b418695b7fc04050e4ab278642f2f26875eddef}} [[phab:T322205|T322205]]

=== 2023-01-24 ===
* 18:52 Rook: remove unused lighthttpd.conf {{Gerrit|0c92944008e9b17426c7bd08e25946b0a175f8d9}} [[phab:T322064|T322064]]
* 12:44 Rook: removing unneeded lines {{Gerrit|897e64e68613f201bfacbd4fe3f1b25be0c219d6}} [[phab:T322742|T322742]]
* 10:54 Rook: bump certifi version {{Gerrit|ee13069fcf84bfa9fa2fedc519bea9a9639c5298}} [[phab:T327764|T327764]]

=== 2023-01-23 ===
* 16:21 Rook: remove automatic update for pywikibot {{Gerrit|029126aea467129de4d6aeb4981c938adc0a48e3}} [[phab:T327668|T327668]]
* 13:09 Rook: remove missed dbProxy entries {{Gerrit|7c52b75f8e00a2af2fd7d75897a959ab841e4d70}} [[phab:T327414|T327414]]

=== 2023-01-19 ===
* 15:55 Rook: cpu limit for renderer {{Gerrit|c48dec0e2d7555169c1faf7e07bf7032ecaa5785}} [[phab:T327306|T327306]]

=== 2023-01-17 ===
* 15:24 Rook: move to magnum {{Gerrit|0eae9000de69fd4d490cb4d87b8c724d11a21350}} [[phab:T326554|T326554]]

=== 2023-01-12 ===
* 14:09 Rook: Variables for nbserve and renderer resource requests {{Gerrit|4eada3f1769a0d48c3bbd1d9a52e3ee8b95e907e}} [[phab:T326723|T326723]]

=== 2023-01-11 ===
* 12:12 Rook: variablize nfs entries for paws-dev {{Gerrit|78c50a9b3fb836b473351bdabb20607199234262}} [[phab:T326631|T326631]]

=== 2023-01-10 ===
* 17:37 Rook: setup dns for paws-dev {{Gerrit|837395a6695d11ae9026df31dd57b25df02102b1}} [[phab:T326629|T326629]]

=== 2023-01-04 ===
* 12:50 Rook: ingress options for ingress-nginx {{Gerrit|4b753ffd22bac5c8ed5085feb4d4e93de2e31e61}} [[phab:T325746|T325746]]

=== 2023-01-03 ===
* 18:45 Rook: update admin users {{Gerrit|ad540ad96af1217d450f8a25cb5427b2e2ad6afd}} [[phab:T326165|T326165]]
* 18:45 Rook: upgrade jupyterhub  {{Gerrit|52027e5fe05e84f69852e5b6ff5fc965fa4ebfc5}} [[phab:T325812|T325812]]

=== 2022-12-16 ===
* 16:08 dcaro: removing coe cluster rook3 ([[phab:T325373|T325373]])

=== 2022-12-13 ===
* 12:48 Rook: bump pywikibot version {{Gerrit|fea0a1a38c8d3a5e2cd29f8fa854664dd7cb6643}} [[phab:T324404|T324404]]

=== 2022-12-05 ===
* 19:40 Rook: Do not load index files automatically {{Gerrit|cd85e584824c983ac5eef6fe1a6010d69c4c8003}} [[phab:T296870|T296870]]

=== 2022-12-01 ===
* 08:30 taavi: root@paws-k8s-control-1:~# for cert in etcd-server etcd-peer etcd-healthcheck-client; do kubeadm certs renew $cert; done # [[phab:T324178|T324178]]

=== 2022-11-28 ===
* 15:43 Rook: drop hostpath [[phab:T321886|T321886]] {{Gerrit|ce2b1f59101e5eb978c2c04d138e704f1363e9fb}}

=== 2022-11-02 ===
* 13:17 Rook: [[phab:T322063|T322063]] Adding commentary regarding nginx rewrites {{Gerrit|b11757be2cd086f2846c2ae154a04a45aa67a7af}}

=== 2022-10-31 ===
* 11:52 Rook: jupyterhub to 3.0 [[phab:T318271|T318271]] {{Gerrit|dc95efa69e1e9887daaa6320dcbd4acc94f624de}}

=== 2022-10-27 ===
* 09:10 Rook: [[phab:T319368|T319368]] openrefine bump {{Gerrit|fe94bc1be2eaca1f87e16f2e8ef685aba46cc4d9}}

=== 2022-10-24 ===
* 15:06 Rook: [[phab:T318273|T318273]] Upgrade jupyterlab {{Gerrit|b342570b6bf55ebc5b3965f7d1fffa5cb4d06157}}

=== 2022-10-20 ===
* 09:57 Rook: autoscaling for renderer deployment [[phab:T320776|T320776]]   {{Gerrit|910e3ca0dbf2a2b9019ff3d86520565e4548d778}}

=== 2022-10-17 ===
* 10:04 Rook: update favicon to paws logo [[phab:T198012|T198012]] {{Gerrit|4019bd87bc9ec588a35944325569177962ccc24d}}
* 05:09 Rook: readme for paws-hub {{Gerrit|1d125c2b88f1c479eb9e1a9250716b49fd4c1e2d}} [[phab:T308949|T308949]]
* 04:40 Rook: update favicon on hub image [[phab:T198012|T198012]] {{Gerrit|146effd07d2adda41d1c1dc9ff5827fae0012ef4}}

=== 2022-10-10 ===
* 17:02 Rook: Bump pywikibot to 7.7.1 {{Gerrit|24d0e2223175024bd11465decbf0a1087c1cc2c3}} [[phab:T320432|T320432]]
* 12:13 Rook: [[phab:T318279|T318279]] Tidy R config files {{Gerrit|c220d165ea779946fa7a3ff98cf806f023e46109}}
* 05:58 Rook: Upgrade IRkernel {{Gerrit|fd62fefdcd42ced29f30be56f82f14ed765fe780}} [[phab:T318275|T318275]]

=== 2022-10-05 ===
* 10:40 dcaro: extended the /srv volumes for both prometheus nodes to 15G to give some space for prometheus to shuffle data (the tsdb size is set to 10G, but it needs a bit more space than that)
* 10:18 arturo: aborrero@paws-k8s-control-1:~$ sudo -i kubectl -n prod rollout restart deployment/proxy ([[phab:T319366|T319366]])
* 10:02 dcaro: checking the prometheus instance for mounts from labstore
* 10:01 dcaro: checking db-backup for labstore mounts (it should have according to hiera)
* 10:00 dcaro: note that ingress were not mounting labstore already
* 10:00 dcaro: removing labstore mounts on ingress-4
* 09:58 dcaro: removing labstore mounts on ingress-3
* 09:57 dcaro: removing labstore mounts on acme-chief
* 09:56 dcaro: removing labstore mounts on control-3
* 09:54 dcaro: removing labstore mounts on control-2

=== 2022-10-03 ===
* 14:05 Rook: remove labstore references #206  [[phab:T317881|T317881]] {{Gerrit|f891319851f2ad6e52a4e3000ea3a2341e262e45}}

=== 2022-09-27 ===
* 07:22 Rook: Upgrade jupyterhub chart #212  [[phab:T311266|T311266]] {{Gerrit|8de422c43525647cf839bbc3e82930626864fa73}}

=== 2022-09-26 ===
* 13:56 taavi: restart the 6 singleuser pods that don't have the new dumps mount points attached yet [[phab:T317144|T317144]]
* 12:42 Rook: Upgrade julia to 1.8.1 #210  {{Gerrit|51307a7d45ed651b4d27763559f3245d72d1aa5c}} [[phab:T318276|T318276]]
* 11:15 Rook: bump pywikibot version #211  [[phab:T318519|T318519]]   {{Gerrit|b540c81d134f1e24c9053d76bd3526186fa15bf6}}
* 08:42 Rook: Remove unused file #209  [[phab:T318277|T318277]] {{Gerrit|9f5034e4c72e5acbad1776bde19042eb9e9f36af}}

=== 2022-09-20 ===
* 18:43 Rook: Bump oauthlib from 3.2.0 to 3.2.1 in /images/minesweeper #207  {{Gerrit|dcfcfaa95da0ee89c50d229d369fcebe0a50232c}}

=== 2022-08-31 ===
* 14:10 Rook: updating openrefine install method to unblock nbclassic [[phab:T312251|T312251]] {{Gerrit|7aa340ca2643e90f9aa3534c6557c142847d1560}}
* 11:44 Rook: jupyterhub version bump [[phab:T316309|T316309]] {{Gerrit|5308be406ca5cdb0bfa823c7040e92fa19c191c9}}

=== 2022-08-29 ===
* 18:48 Rook: jupyterlab version bump {{Gerrit|0b2532f0590fc5c1ae5b5f5a7148bdf9f4b1f92c}} [[phab:T308042|T308042]]

=== 2022-08-25 ===
* 15:05 Rook: OpenRefine version bump [[phab:T316218|T316218]]  {{Gerrit|55271d2fe80a04f7b2b17426a882f81b4ca8a413}}
* 09:54 Rook: resource requests for renderer {{Gerrit|3598be38693668e1733c34fc6df13318db74f17e}} [[phab:T315682|T315682]]

=== 2022-08-24 ===
* 13:44 Rook: Upgrade ingress-nginx to 1.3.0 {{Gerrit|8e6b577fbc8a48167919dd20365967d68eb60a7d}}

=== 2022-08-23 ===
* 11:40 Rook: setup requests for nbserve [[phab:T315670|T315670]] {{Gerrit|e558ee7f619a590838e106c9866e2f8ebae33e58}}

=== 2022-08-22 ===
* 12:40 Rook: Upgrade pywikibot 7.5.0 -&gt; 7.6.0 #193  {{Gerrit|456d3f2fe0}} [[phab:T315745|T315745]]
* 11:44 Rook: updating key method #190 {{Gerrit|5631062a8a}} [[phab:T312096|T312096]]

=== 2022-08-04 ===
* 12:26 Rook: resource requests for quota {{Gerrit|cc8556d3ae422b99bb60c26d6ac0ce6f1e285b07}} [[phab:T314432|T314432]]

=== 2022-08-02 ===
* 12:22 Rook: deleting worker-1 from cluster and shutting down [[phab:T313287|T313287]]

=== 2022-07-25 ===
* 20:25 Rook: [[phab:T313728|T313728]] #188 revert nbclassic to restore openrefine
* 11:18 Rook: {{Gerrit|9336306f7be6791687f411dcd1db6735b670f2c9}} update pywikibot

=== 2022-07-22 ===
* 15:43 Rook: updating to give hub pod a cpu request so that it cannot be resource starved

=== 2022-07-19 ===
* 08:12 taavi: drain paws-k8s-worker-1 [[phab:T313287|T313287]]

=== 2022-07-14 ===
* 14:21 Rook: {{Gerrit|c6ef2b22b1ab3671d3f463fb2a0fa67aec265caa}} update nbclassic [[phab:T312251|T312251]]

=== 2022-07-13 ===
* 18:25 Rook: {{Gerrit|002d1a8261cfb701a64959d7e53c876d9a63393e}} update dockerfile to give julia and openrefine sections
* 11:59 Rook: {{Gerrit|cd8ec3987350e71e3798bfbcea0e65a9d5fa6b52}} node from its own repo #182

=== 2022-07-12 ===
* 13:12 Rook: {{Gerrit|2e11fb076f6b3cda1b538f58d022bf23e417361a}} libssl from repo #181 [[phab:T312236|T312236]]

=== 2022-07-11 ===
* 14:09 Rook: refactor: create R install section #180 {{Gerrit|bac49ef0f2a2daeb0d5329c4f64d6c4591c2daa0}} [[phab:T312570|T312570]]
* 11:15 Rook: {{Gerrit|08f1e33a8f408275f257c0911c1fb4f991e70f1e}} reorder minimal install #179 [[phab:T312593|T312593]]

=== 2022-07-07 ===
* 20:56 Rook: SPARQL to own section of build {{Gerrit|d3827e3fc6a137e54c997b4b935d9583b9a0e9b6}} [[phab:T312569|T312569]]
* 14:26 Rook: update pwb version {{Gerrit|6fcb349aef861ee229ce856672720586223dd9d4}} [[phab:T311958|T311958]]
* 11:17 Rook: unblock singleuser container build [[phab:T312088|T312088]] {{Gerrit|977c8d4dc6f38be96f29914ee0cafbb7d837b06d}}

=== 2022-06-27 ===
* 16:42 Rook: Make renderer accessable in dev env #172 {{Gerrit|9f2f171ff2a519fa11a784e8236eaf337bb13d08}}  [[phab:T308986|T308986]]

=== 2022-06-23 ===
* 13:20 Rook: remove deployhook token #171 [[phab:T310382|T310382]] {{Gerrit|74180f43cf6487f53b4812106ab8471aba2f8528}}

=== 2022-06-21 ===
* 12:36 Rook: [[phab:T302164|T302164]] #173 upgrading single user container {{Gerrit|b687ca6aedc745c63b2659124dca0dab01d38173}}

=== 2022-06-15 ===
* 18:03 Rook: removing unnecessary yaml #167 {{Gerrit|f4733023a1b63e48d12f99da5344af5513aea245}}
* 17:04 Rook: Move renderer to ubuntu container #168 {{Gerrit|078808991a683f44564a1dd8b3d13e9c227617c7}}

=== 2022-06-14 ===
* 13:11 Rook: Removing leftover db-proxy bits #170 {{Gerrit|8795bf7ded5eb2406a7dce70d8bd4450f6b2511d}}

=== 2022-06-07 ===
* 13:01 Rook: remove db-proxy #166 [[phab:T309794|T309794]]  {{Gerrit|643e112be91c355b8769caeab9e814dea9172b7d}}

=== 2022-06-06 ===
* 13:54 Rook: [[phab:T308975|T308975]] Move away from git to pip for nbconvert #162 {{Gerrit|54f93d02f62ec98d789a2e64a1288dff763833db}}
* 12:21 Rook: [[phab:T308926|T308926]] Move nbserve to upstream container #165 {{Gerrit|1767870b6724e863e705e86bf85c9e0bb90bafdf}}

=== 2022-06-02 ===
* 17:02 Rook: scaling db-proxy to zero [[phab:T309794|T309794]]

=== 2022-05-22 ===
* 12:59 Rook: deploying [[phab:T308862|T308862]]: Fix version of nbvconvert #158 {{Gerrit|47a7db6b320e70a4183b0a1891096ec5b00129fe}}

=== 2022-05-18 ===
* 12:55 Rook: bump jupyterhub version #149 [[phab:T308568|T308568]] {{Gerrit|41f03a544041318f1fad479b32ae46ac9e816a55}}
* 11:10 Rook: updating ingress from v1beta1 #134 [[phab:T294342|T294342]] {{Gerrit|5107562ccaf1160ba09a62ac7e2de9cd5e90f79f}}

=== 2022-05-16 ===
* 09:36 dcaro: restarted reload-acme-chief-backend.service to ensure certs are refreshed

=== 2022-05-14 ===
* 16:16 andrewbogott: restarting acme-chief.service on paws-acme-chief-01 for [[phab:T308383|T308383]]

=== 2022-05-11 ===
* 13:44 Rook: update pwb version and pin jupyterlab version {{Gerrit|ef3e38c671e20fad6b8297dfb20a14a913a8dda5}}

=== 2022-05-10 ===
* 13:56 Rook: upgrade pywikibot on container start {{Gerrit|437f46a1caf56cf860f85211af4567735410a621}}

=== 2022-04-27 ===
* 17:02 Rook: pywikibot version bump {{Gerrit|3c42a6243161ce6f6d8ff39829a58fd576d161b5}}

=== 2022-04-18 ===
* 17:29 Rook: updating links to phab with prefilled ticket links {{Gerrit|aef7c671a69a66a9872a48a24169f1f6bf2ffc4f}}
* 12:30 Rook: update pywikibot {{Gerrit|6db74b6c866021686a49aa8a2c4eb2d5da3bdddb}}

=== 2022-04-16 ===
* 18:36 wm-bot: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics ({{Gerrit|2c485e9}}) - cookbook ran by taavi@runko

=== 2022-04-04 ===
* 12:31 taavi: moving all VMs from paws-puppetmaster-01 -&gt; paws-puppetmaster-2

=== 2022-03-29 ===
* 10:36 Rook: upgrading pywikibot {{Gerrit|702f21dfe79e558d504c34474b295f1b8a5f75bb}}

=== 2022-03-21 ===
* 11:11 Rook: deploying jupyterlab {{Gerrit|cd6ee196a66c8a7d48504166fd776973054535a4}}

=== 2022-03-10 ===
* 12:23 Rook: updating banner to note ui will update soon {{Gerrit|462ab184e59a7f791bbc8790e823146f6e482920}}

=== 2022-03-08 ===
* 13:26 Rook: upgrading open refine {{Gerrit|c116d64f07dce58e884be64d0cdee70c79739dd4}}

=== 2022-03-07 ===
* 11:12 Rook: deploying paws realtime collaboration {{Gerrit|246e2afa97060dbd010b5a36ac88589f972d741f}}

=== 2022-03-02 ===
* 14:20 Rook: deploying fixed version of jupyter-rsession-proxy {{Gerrit|abe89f6a70d75492c1ad1baa2bc955204bce3c2e}}

=== 2022-03-01 ===
* 13:38 Rook: deploying pyaudio fix {{Gerrit|978fb648dbd1d1f351bba74b6e2aff4023137087}}

=== 2022-02-23 ===
* 13:13 Rook: deploying {{Gerrit|e6eedbc58bd6f1f074912f4faf8075275ae13819}} cleanup

=== 2022-02-15 ===
* 18:20 chicocvenancio: added psp for minesweeper
* 16:04 mdipietro: updating pywikibot {{Gerrit|2fc27c99db35515121f5b80c9f3df5e648e56b58}}
* 14:21 chicocvenancio: Deploying minesweeper

=== 2022-01-25 ===
* 14:30 mdipietro: deployed {{Gerrit|93d33c44a876fda8928719353ac0d5cf75651fd7}} PR122

=== 2021-12-28 ===
* 14:14 taavi: deploying https://github.com/toolforge/paws/pull/121

=== 2021-12-20 ===
* 18:20 majavah: deploying calico v3.21.0 ([[phab:T292698|T292698]])

=== 2021-12-16 ===
* 12:55 mdipietro: deploying {{Gerrit|dcac479b8c68631977441be5aaff8f6335659e1a}} [[phab:T294431|T294431]]

=== 2021-12-02 ===
* 19:12 chicocvenancio: deploy PR 111 [[phab:T295257|T295257]]
* 12:52 mdipietro: upgrading pywikibot {{Gerrit|0f5d28d23f49f9ba4008d48f9ff8323dd48c702b}}

=== 2021-12-01 ===
* 11:36 mdipietro: deploying lsof pr-76 {{Gerrit|a378845f87ff57427b44b8a8f4b616a0a74cfbf4}}

=== 2021-11-29 ===
* 13:02 chicocvenancio: deploy PR 113 [[phab:T295761|T295761]]
* 12:29 chicocvenancio: deploy PR 112 [[phab:T295761|T295761]]

=== 2021-11-25 ===
* 21:37 chicocvenancio: rollback singleuser to PR #96  [[phab:T295257|T295257]]
* 21:15 chicocvenancio: deploy PR #110 changing singleuser to bump openrefine version [[phab:T295257|T295257]]

=== 2021-11-23 ===
* 14:19 mdipietro: increased cull timeout with deploy of {{Gerrit|3e5726489d370db3a11f166d90190f6271e5a743}}

=== 2021-11-22 ===
* 12:57 mdipietro: added julia to paws with {{Gerrit|7b58fb09292a27f939f93ffba6ff00f8028302f2}}
* 11:04 mdipietro: added julia to paws with {{Gerrit|12bfdad0b4d7c60737b79914b6098029235ce9ef}}

=== 2021-11-11 ===
* 08:35 majavah: disabling pod preset controller in preparation for [[phab:T291913|T291913]]

=== 2021-11-09 ===
* 16:24 mdipietro: deployed PR97 ({{Gerrit|85c085f5309d45c598c3089fd64fa4810c769c06}}) Update Pywikibot to 6.6.2

=== 2021-11-03 ===
* 12:41 mdipietro: deployed https://github.com/toolforge/paws/pull/92 ({{Gerrit|4961101972be6f27c1c96e327a82457346965f32}}) [[phab:T150098|T150098]]

=== 2021-11-01 ===
* 12:31 majavah: upgrade ingress-nginx [[phab:T292771|T292771]]

=== 2021-10-28 ===
* 14:35 chicocvenancio: set team toolforge/wmcsadmins as maintainers for github repo

=== 2021-10-26 ===
* 15:06 chicocvenancio: delete orphan pods for 2 users

=== 2021-10-22 ===
* 16:38 majavah: deploy https://github.com/toolforge/paws/pull/91

=== 2021-10-21 ===
* 12:58 mdipietro: upgraded to {{Gerrit|923250f7cd0b522259abdad450fbd2dfb16357bf}} which was really not an upgrade as the diff gave nothing. Though now it is clear what is deployed.

=== 2021-09-07 ===
* 22:14 bstorm: upgraded k8s to 1.19.13 [[phab:T287399|T287399]]

=== 2021-08-18 ===
* 19:09 bstorm: redeployed hub with trove database backend instead of toolsdb

=== 2021-07-29 ===
* 14:09 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]

=== 2021-07-25 ===
* 16:09 majavah: deleting ingress pod running on worker-6 to get it to re-appear in ingress-4

=== 2021-07-21 ===
* 19:53 bstorm: deployed new maintain-kubeusers [[phab:T285011|T285011]]
* 19:53 bstorm: deployed new rbac for maintain-kubeusers changes [[phab:T285011|T285011]]
* 16:59 majavah: deploying calico v3.18.4 [[phab:T280342|T280342]]
* 15:52 majavah: add my key to passwords::root::extra_keys
* 15:00 majavah: starting kubernetes upgrades [[phab:T280302|T280302]]

=== 2021-07-14 ===
* 10:38 majavah: correction: undeploy old ingress [[phab:T264221|T264221]]
* 10:35 majavah: undeploy old ingress [[phab:T266050|T266050]]

=== 2021-07-13 ===
* 07:51 majavah: renewing tools-prometheus certificates

=== 2021-07-12 ===
* 13:18 majavah: ingress upgrade completed
* 13:05 majavah: moving user traffic to updated ingress-nginx [[phab:T264221|T264221]]

=== 2021-07-01 ===
* 12:04 majavah: deploy ingress-nginx 0.46 via the helm chart to paws [[phab:T264221|T264221]]

=== 2021-06-30 ===
* 20:05 bstorm: tried force delete on the ingress-nginx-gen2 namespace, which doesn't appear to be working either until metrics-server is fixed [[phab:T285905|T285905]]
* 20:00 bstorm: renewed k8s metrics-server certs and the deployment
* 18:04 majavah: renew kubernetes metrics-server certificate
* 17:26 majavah: creating paws-k8s-ingress-[3-4] and joining them to the k8s cluster [[phab:T264221|T264221]]
* 17:16 bstorm: temporarily increased quota to 60 cores to enable [[phab:T264221|T264221]]

=== 2021-06-03 ===
* 20:43 chicocvenancio: tagged new singleuser image, fixes [[phab:T283969|T283969]]

=== 2021-05-27 ===
* 21:53 bstorm: added paws-k8s-control-2.paws.eqiad.wmflabs back to the list of control nodes at the proxy
* 21:50 bstorm: renewed the certs for paws-k8s-control-2
* 20:37 bstorm: removed paws-k8s-control-2.paws.eqiad.wmflabs from the proxy because it is somewhat broken (certs expired)
* 19:41 bstorm: forced removal of openrefine in paws for now and deleted all current user server pods to force use of the new image

=== 2021-05-23 ===
* 01:45 bstorm: manually pulling updated singleuser images because I hate waiting to test things
* 01:36 bstorm: update latest tag for singleuser image to {{Gerrit|b894471}} for https://github.com/toolforge/paws/pull/69
* 00:56 bstorm: deploying changes for https://github.com/toolforge/paws/pull/67

=== 2021-05-21 ===
* 00:06 bstorm: creating trove mysql instance pawsdb-1 [[phab:T267683|T267683]]

=== 2021-05-12 ===
* 19:33 bstorm: added taavi to paws.admin

=== 2021-05-11 ===
* 09:17 Majavah: set `profile::wmcs::kubeadm::docker_vol: false` on ingress nodes [[phab:T282087|T282087]]
* 09:15 arturo: added user `taavi` (Majavah) as projectadmin

=== 2021-04-20 ===
* 23:18 bstorm: deploying updated paws-hub image for https://github.com/toolforge/paws/pull/63

=== 2021-04-02 ===
* 21:50 bstorm: deploying latest PRs to add a note on the wikireplicas changes

=== 2020-12-21 ===
* 20:27 bstorm: applied tuning for timeouts and elections on the k8s etcd pods of 300 for heartbeat and 3000 for elections [[phab:T267966|T267966]]

=== 2020-12-17 ===
* 02:22 bstorm: Set PAWS hub back to using mariadb [[phab:T266587|T266587]]

=== 2020-12-16 ===
* 18:21 chicocvenancio: move paws to sqlite while toolsdb is down.

=== 2020-12-10 ===
* 17:00 arturo: fixing /etc/kubernetes/kublet.conf and restarting kubelet in paws-k8s-control-1 ([[phab:T269865|T269865]])

=== 2020-12-05 ===
* 00:42 bd808: `kubectl delete po renderer-794886b9cd-9nc6c -n prod` after seeing lots of listen queue full errors in the pod logs.

=== 2020-11-30 ===
* 18:22 bstorm: 1.17 upgrade for kubernetes complete [[phab:T268669|T268669]]
* 17:25 bstorm: upgrading the worker nodes (this will likely kill services briefly when some pods are rescheduled) [[phab:T268669|T268669]]
* 17:14 bstorm: updated the calico-kube-controllers deployment to use our internal registry to deal with docker-hub rate-limiting [[phab:T268669|T268669]] [[phab:T269016|T269016]]
* 17:09 chicocvenancio: delete orphaned jupyter server pod `kubectl -n prod delete pod jupyter--45volutionoftheuniverse`. Respective server not running in jupyter admin UI.
* 16:31 bstorm: upgrading pods on paws-k8s-control-3 [[phab:T268669|T268669]]
* 16:17 bstorm: starting upgrade on paws-k8s-control-2 [[phab:T268669|T268669]] (first kubectl drain paws-k8s-control-2 --ignore-daemonsets)
* 15:53 bstorm: proceeding with upgrade to 1.17 on paws-k8s-control-1 [[phab:T268669|T268669]]
* 15:49 bstorm: draining paws-k8s-control-1 for upgrade [[phab:T268669|T268669]]
* 12:49 arturo: disable puppet in all k8s nodes to prepare for the upgrade ([[phab:T268669|T268669]])
* 12:49 arturo: set hiera `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-17'` at project level ([[phab:T268669|T268669]])

=== 2020-11-16 ===
* 22:13 bstorm: deploying new paws changes for multiinstance readiness

=== 2020-11-10 ===
* 20:16 chicocvenancio: restart hub to apply move to sqlite.  [[phab:T267667|T267667]]
* 16:41 arturo: set paws in sqlite mode because [[phab:T266587|T266587]] (kubectl --namespace prod edit configmap hub-config)

=== 2020-10-15 ===
* 19:12 andrewbogott: uncordoned paws-k8s-worker-1 and -2
* 18:48 andrewbogott: draining paws-k8s-worker-2 for move to ceph
* 18:36 andrewbogott: draining paws-k8s-worker-1 for move to ceph

=== 2020-09-29 ===
* 10:59 arturo: last 2 commands should help puppet agent in the paws project, previously it had issues fetching acme-chief certs because an API update
* 10:58 arturo: aborrero@paws-acme-chief-01:~$ sudo systemctl restart uwsgi-acme-chief.service
* 10:56 arturo: aborrero@paws-acme-chief-01:~$ sudo systemctl restart acme-chief.service

=== 2020-08-14 ===
* 17:09 bstorm: backing up the old proxy config to NFS and deleting paws-proxy-02 [[phab:T211096|T211096]]

=== 2020-08-07 ===
* 22:30 bstorm: removing downtime for paws and front page monitor [[phab:T211096|T211096]]
* 18:01 bstorm: shutting down paws-proxy-02 [[phab:T211096|T211096]]
* 17:05 bstorm: running the final rsync to the new cluster's nfs [[phab:T211096|T211096]]
* 16:08 bstorm: changing paws.wmflabs.org to point at the new cluster ip 185.15.56.57 [[phab:T211096|T211096]]
* 16:02 bstorm: LAST MESSAGE WRONG: switching NEW cluster to toolsdb [[phab:T211096|T211096]]
* 16:02 bstorm: switching old cluster to toolsdb [[phab:T211096|T211096]]
* 15:58 bstorm: switching old cluster to sqlite [[phab:T211096|T211096]]
* 15:53 bstorm: downtiming alerts in case they need changes (seems likely) [[phab:T211096|T211096]]

=== 2020-07-30 ===
* 20:40 bstorm: upgrading the singleuser image to test shuffling around some of the pip installs
* 16:38 bstorm: removing the *.paws.wmflabs.org SNI name because it won't be used and it might trigger a re-issue of certs [[phab:T255249|T255249]]
* 15:39 bstorm: upgrading acme-chief to 0.27-1

=== 2020-07-29 ===
* 18:03 bstorm: powering on paws-k8s-haproxy-1 because that worked fine
* 18:00 bstorm: powering off paws-k8s-haproxy-1 to test failover

=== 2020-07-24 ===
* 17:25 bstorm: to force repulling of every image everywhere, uninstalling paws in the new cluster and reinstalling it [[phab:T258812|T258812]]
* 09:39 arturo: dropped the DNS wildcard record `*.paws.wmcloud.org IN A 185.15.56.57` and created concrete CNAME records for the FQDNs we actually use ([[phab:T211096|T211096]])

=== 2020-07-23 ===
* 22:51 bstorm: deploying via the default 'latest' tag in the new cluster [[phab:T211096|T211096]]
* 22:48 bstorm: tagged the newbuild tags with &quot;latest&quot; to set sane defaults for all images in the helm chart [[phab:T211096|T211096]]
* 21:14 bstorm: pushing quay.io/wikimedia-paws-prod/nbserve:newbuild to main repo [[phab:T211096|T211096]]
* 21:11 bstorm: pushing quay.io/wikimedia-paws-prod/deploy-hook:newbuild to main repo [[phab:T211096|T211096]]
* 21:09 bstorm: pushing quay.io/wikimedia-paws-prod/singleuser:newbuild to the main repo [[phab:T211096|T211096]]
* 21:08 bstorm: pushing quay.io/wikimedia-paws-prod/paws-hub:newbuild to the main repo [[phab:T211096|T211096]]
* 21:06 bstorm: pushing dbproxy docker image for new cluster into main quay.io repo [[phab:T211096|T211096]]

=== 2020-07-22 ===
* 23:32 bstorm: setting the default NFS version to 4.2 while excepting the two stretch servers [[phab:T257945|T257945]]

=== 2020-07-21 ===
* 15:13 chicocvenancio: merge pr #50 to fix [[phab:T258142|T258142]]

=== 2020-07-06 ===
* 21:41 bstorm: deployed ingress to redirect paws.wmcloud.org to the wikitech doc page [[phab:T195217|T195217]]

=== 2020-06-30 ===
* 23:00 bstorm: added paws-public.wmflabs.org to the alt-names for acme-chief, which broke it until we hand off the zone to the paws project &lt;sorry!&gt; [[phab:T195217|T195217]] [[phab:T255997|T255997]]

=== 2020-06-26 ===
* 21:57 bstorm: applied the metrics manifests to kubernetes to enable metrics-server, cadvisor, etc. [[phab:T256361|T256361]]

=== 2020-06-25 ===
* 22:52 bstorm: created paws-k8s-worker-5/6/7 as x-large nodes to bring the cluster up to roughly the same capacity as the existing one using soft anti-affinity [[phab:T211096|T211096]] [[phab:T253267|T253267]]
* 22:43 bstorm: bumped quota up to 24 instances, 128 GB RAM and 56 cores [[phab:T211096|T211096]]
* 16:39 bstorm: deleted the deployhook from the in-progress new cluster for now just in case [[phab:T211096|T211096]]
* 15:44 bstorm: deployed a proof-of-concept paws-public setup in the new cluster [[phab:T255997|T255997]]

=== 2020-06-24 ===
* 23:18 bstorm: added A record for *.paws.wmcloud.org to public and hub to use [[phab:T211096|T211096]] [[phab:T255997|T255997]] [[phab:T195217|T195217]]
* 21:45 bstorm: doing an initial rsync of the paws userhomes to the new project [[phab:T160113|T160113]]

=== 2020-06-19 ===
* 10:01 arturo: enabled `paws.wmflabs.org` and `*.paws.wmflabs.org` as valid ingress domains (acme-chief TLS cert, haproxy, etc) ([[phab:T195217|T195217]])

=== 2020-06-17 ===
* 21:51 bstorm_: upgraded chart in the new cluster to include resource limits [[phab:T251298|T251298]]
* 21:51 bstorm_: upgraded chart in the new cluster to include resource limits

=== 2020-06-16 ===
* 15:48 arturo: change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 ([[phab:T195217|T195217]])
* 15:47 arturo: associate floating IP 185.15.56.57 with haproxy VIP port ([[phab:T295217|T295217]])
* 15:43 arturo: allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 1b40be58-7182-41aa-95ce-797f94f83d66` ([[phab:T295217|T295217]])
* 15:43 arturo: allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 9ccc43d9-1a8a-4287-afda-67e8bab27a9f` ([[phab:T295217|T295217]])
* 15:37 arturo: `aborrero@cloudcontrol1004:~ 1 $ sudo wmcs-openstack --os-project-id=paws port create --network 7425e328-560c-4f00-8e99-{{Gerrit|706f3fb90bb4}} paws-haproxy-vip` ([[phab:T295217|T295217]])
* 15:23 arturo: live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for [[phab:T195217|T195217]]

=== 2020-06-15 ===
* 15:59 arturo: created DNS record `deploy-hook.paws.wmcloud.org IN CNAME paws.wmcloud.org` ([[phab:T195217|T195217]])
* 12:28 arturo: manually created an Ingress object to test routing to the hub ([[phab:T195217|T195217]])
* 12:20 arturo: created DNS record `paws.wmcloud.org IN A 185.15.56.57` ([[phab:T195217|T195217]])
* 12:19 arturo: associate floating IP 185.15.56.57 with VM paws-k8s-haproxy-1 ([[phab:T195217|T195217]])
* 12:18 arturo: release floating IP not in use: 185.15.56.42
* 12:18 arturo: release floating IP not in use: 185.15.56.43
* 11:45 arturo: reset wikitech user password for the service account `paws-dns-manager` to what is in labs/private.git/hieradata/common.yaml `profile::acme_chief::cloud::designate_sync_password` ([[phab:T195217|T195217]])

=== 2020-06-12 ===
* 18:49 bstorm_: deployed a test of paws chart in the new cluster [[phab:T211096|T211096]]
* 13:23 arturo: assigned the DNS zone `paws.wmcloud.org` ([[phab:T195217|T195217]])
* 13:13 arturo: live-hacking session in the puppetmaster ended
* 13:05 arturo: live-hacking puppet tree in paws-puppetmaster-01 for [[phab:T195217|T195217]]
* 11:55 arturo: `aborrero@cloudcontrol1004:~ $ sudo wmcs-openstack role add --user paws-dns-manager --project paws observer` ([[phab:T255252|T255252]])
* 11:55 arturo: `aborrero@cloudcontrol1004:~ $ sudo wmcs-openstack role add --user paws-dns-manager --project paws designateadmin` ([[phab:T255252|T255252]])
* 11:51 arturo: created service account `paws-dns-manager` in wikitech ([[phab:T255252|T255252]])
* 11:31 arturo: introduced acme-chief private data into labs/private in paws-puppetmaster-01 ([[phab:T255252|T255252]])
* 11:02 arturo: created puppet prefix 'paws-acme-chief' ([[phab:T255252|T255252]])
* 11:01 arturo: created VM paws-acme-chief-01 ([[phab:T255252|T255252]])

=== 2020-06-11 ===
* 11:11 arturo: deployed nginx-ingress for some early testing (not definitive) with code https://github.com/crookedstorm/paws/commit/bee62b3fd57f9804aa27e7b8b41fde50bd93df94 ([[phab:T195217|T195217]])
* 10:15 arturo: added role (just a label) for ingress nodes: `kubectl label node paws-k8s-ingress-1 kubernetes.io/role=ingress` ([[phab:T195217|T195217]])

=== 2020-06-04 ===
* 14:16 arturo: added node taints to ingress nodes: `kubectl taint nodes paws-k8s-ingress-1 ingress=true:NoSchedule` ([[phab:T195217|T195217]])
* 12:18 arturo: bootstrapped paws-k8s-ingress nodes, added them to the k8s cluster ([[phab:T195217|T195217]])
* 12:04 arturo: created `paws-k8s-ingress` puppet prefix and add the `role::wmcs::paws::k8s::worker` role ([[phab:T195217|T195217]])
* 12:02 arturo: created 2 medium VM instances: paws-k8s-ingress-1 and paws-k8s-ingress-2 with haproxy anti-affinity ([[phab:T195217|T195217]])

=== 2020-05-26 ===
* 22:34 bstorm_: restored the deployment for maintain-kubeusers so anyone added to the paws.admin group will have admin on the cluster now that the bug is fixed [[phab:T211096|T211096]] [[phab:T246059|T246059]]
* 22:05 bstorm_: temporarily deleted the deployment for maintain-kubeusers pending patch to fix context creation for new admin accounts [[phab:T211096|T211096]] [[phab:T246059|T246059]]
* 22:04 bstorm_: created paws-focused PodSecurityPolicies and the prod namespace in the new cluster [[phab:T211096|T211096]]
* 22:03 bstorm_: created paws.admin group and kubernetes admin accounts on the new k8s cluster [[phab:T211096|T211096]] [[phab:T246059|T246059]]
* 18:29 bstorm_: bootstrapped the new control plane nodes [[phab:T211096|T211096]]
* 15:27 bstorm_: updated profile::wmcs::kubeadm::kubernetes_version to 1.16.10 for cluster init [[phab:T211096|T211096]]

=== 2020-05-21 ===
* 23:04 bstorm_: added profile::wmcs::kubeadm::k8s::encryption_key and profile::wmcs::kubeadm::k8s::node_token to labs/private [[phab:T211096|T211096]]
* 14:53 bstorm_: adding the hiera values to horizon for bootstrapping k8s [[phab:T211096|T211096]]
* 14:39 arturo: point record `k8s.svc.paws.eqiad1.wikimedia.cloud` to `172.16.1.186` (which is paws-k8s-control-1, for the initial bootstrap) ([[phab:T211096|T211096]])
* 12:48 arturo: created record `k8s.svc.paws.eqiad1.wikimedia.cloud` pointing to `172.16.0.191` (which is paws-k8s-haproxy-1) ([[phab:T211096|T211096]])
* 12:34 arturo: created and transferred DNS zone `svc.paws.eqiad1.wikimedia.cloud` ([[phab:T211096|T211096]])

=== 2020-05-20 ===
* 22:35 bstorm_: created paws-k8s-worker-1/2/3/4 [[phab:T211096|T211096]]
* 22:12 bstorm_: created paws-k8s-haproxy-1/2 with antiaffinity group [[phab:T211096|T211096]]
* 21:36 bstorm_: created paws-k8s-control-1/2/3 with appropriate sec group and server group [[phab:T211096|T211096]]
* 18:59 bstorm_: created anti-affinity group &quot;controlplane&quot; [[phab:T211096|T211096]]
* 16:38 bstorm_: deleting the old shut-down VMs from the last effort to rebuild paws [[phab:T211096|T211096]]
* 16:36 bstorm_: cleaned up the old DNS entries for the external LBs that have been off for a year

=== 2020-03-20 ===
* 14:03 jeh: upgrade paws-puppetmaster-01 to v5 [[phab:T241719|T241719]]

=== 2020-02-14 ===
* 21:31 andrewbogott: restarting paws-puppetmaster-01 so its clients can connect

=== 2020-01-09 ===
* 18:06 bstorm_: rebooting tools-paws-master-01 [[phab:T242353|T242353]]
* 14:28 chicocvenancio: shutdown unused instances

=== 2019-12-13 ===
* 00:27 bstorm_: rebooting the paws master since it is in a bad state after the openstack maintenance as well.

=== 2019-11-01 ===
* 21:15 Krenair: Updated paws-apiserver.wmflabs.org A record list to remove 172.16.2.151 which is not allocated to any instance. The other two A records point to valid instances in the paws project.

=== 2019-10-23 ===
* 09:03 arturo: paws-master-01/03 and a couple of other servers are down because hypervisor is rebooting

=== 2019-10-14 ===
* 22:32 bd808: Removed project member &quot;Afrodric&quot;. Looks like someone added accidentally when trying to make aborrero as project member
* 22:31 bd808: Added Krenair as project member

=== 2019-05-18 ===
* 11:13 chicocvenancio: point paws-proxy-02 to tools-paws-worker-1006 on paws-deploy-hook hostname ([[phab:T218380|T218380]])

=== 2019-04-26 ===
* 15:27 chicocvenancio: paws-proxy-02 redirecting main page to https://www.mediawiki.org/wiki/PAWS [[phab:T221886|T221886]]

=== 2019-04-16 ===
* 17:15 chicocvenancio: move paws-proxy-02 reload nginx
* 17:07 chicocvenancio: move paws-proxy-02 to point to tools-paws-worker-1006 for upcoming master move

=== 2019-03-27 ===
* 23:46 chicocvenancio: moving paws host in `paws-proxy-02`  back to `tools-paws-master-01` [[phab:T219460|T219460]]
* 22:10 chicocvenancio: moving paws host in `paws-proxy-02` to `tools-paws-worker-1005` [[phab:T219460|T219460]]

=== 2019-03-25 ===
* 14:12 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project ([[phab:T211096|T211096]])
* 14:07 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project [[phab:T211096|T211096]]
* 13:54 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project ([[phab:T211096|T211096]])

=== 2019-03-15 ===
* 02:25 gtirloni: activated TLS termination using Let's Encrypt on paws-proxy-02
* 02:25 gtirloni: removed webproxies and created new A records pointing directly to paws-proxy-02

=== 2019-02-21 ===
* 09:22 gtirloni: upgraded and rebooted paws-proxy-02

=== 2019-02-20 ===
* 15:00 andrewbogott: deleting the long-shut-down paws-proxy-01

=== 2019-02-15 ===
* 01:28 bd808: Re-enabled PAWS vhost on paws-proxy-02

=== 2019-02-14 ===
* 22:25 gtirloni: downtimed PAWS in Icinga
* 22:16 gtirloni: Activated maintenance page on paws-proxy-02 nginx config

=== 2019-02-13 ===
* 08:32 arturo: switch paws-proxy-02 puppetmaster to labs-puppetmaster.wikimedia.org

=== 2019-01-24 ===
* 19:20 andrewbogott: shutting down paws-proxy-01
* 19:11 chicocvenancio: moved config, ready to receive  traffic on paws-proxy-02  [[phab:T214613|T214613]]
* 18:34 chicocvenancio: firing up paws-proxy-02 for [[phab:T214613|T214613]]

=== 2019-01-23 ===
* 20:46 chicocvenancio: moving paws_public  proxy_pass to https://172.16.6.39 in paws-proxy-01

=== 2018-10-25 ===
* 23:58 gtirloni: Started tools-paws-worker-1010 ([[phab:T208006|T208006]])

=== 2018-08-03 ===
* 20:19 andrewbogott: deleting paws-master-01 and paws-node-1002; unused

=== 2018-07-03 ===
* 22:49 bstorm_: added stricter image space reclaiming arguments to kubelet

=== 2018-06-20 ===
* 17:39 chicocvenancio: edited paws-proxy-01 to pass http_x_forwarded_proto as it receives [[phab:T197248|T197248]]

=== 2018-05-04 ===
* 02:48 chicocvenancio: killed 25 pods with more than one hour inactivity through admin interface

=== 2018-03-14 ===
* 21:49 chicocvenancio: updated k8s control plane, updating nodes to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-02-23 ===
* 18:33 chicocvenancio: redirected tools.wmflabs.org/paws to paws.wmflabs.org and deleted old k8s ReplicationControllers ([[phab:T188068|T188068]])

=== 2018-02-22 ===
* 22:11 chicocvenancio: ([[phab:T175202|T175202]]) culler is running and killing pods as designed!
* 21:13 chicocvenancio: jupyterhub updated to fix culler ([[phab:T175202|T175202]]) culler already ran without 404
* 17:43 chicocvenancio: manually ran culler inside hub container

=== 2018-02-21 ===
* 17:03 chicocvenancio: deleted query-killer k8s deployment [[phab:T187818|T187818]]

=== 2018-02-16 ===
* 20:18 chicocvenancio: changed userhomes group for [[phab:T185434|T185434]]  workarround

=== 2018-02-15 ===
* 01:10 chicocvenancio: changed group of all userhome folders to tools.paws

=== 2018-02-04 ===
* 12:21 chicocvenancio: changed group of all userhome folders to tools.paws

=== 2017-12-19 ===
* 22:11 bd808: Killed tiller pod that was in crashloopbackoff

=== 2017-09-28 ===
* 21:25 andrewbogott: server docker restart on paws-node-1002; disk is full and docker is holding open a lot of deleted files

=== 2017-03-20 ===
* 21:25 andrewbogott: migrating paws-base-01 to labvirt1013

=== 2016-05-10 ===
* 20:36 bd808: Visit https://foo-pawsbase.wmflabs.org/ to see list of wikis in wikifarm
* 20:32 bd808: Fixed hiera config so https://wikidata-pawsbase.wmflabs.org routes correctly
* 15:47 YuviPanda: added bd808 to project and made as admin

{{SAL|Project Name=paws}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>qq09tt4206esu9gx0wqsbpq9jbnuxu4</sha1>
    </revision>
  </page>
  <page>
    <title>Portal:Cloud VPS/Admin/VM images</title>
    <ns>0</ns>
    <id>442314</id>
    <revision>
      <id>2435521</id>
      <parentid>2332145</parentid>
      <timestamp>2026-07-12T01:34:43Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix [[w:WP:LISTGAP]] issues</comment>
      <origin>2435521</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="20617" sha1="8l0g17zimu0na7385gz1mjvkppx129p" xml:space="preserve">{{Cloud VPS nav}}
Cloud VPS uses special VM images that contain all the customizations required for our environment.

== Builders ==

Images for Debian Buster (and subsequent release) are based off of official debian upstream builds. These are built on a cloudcontrol node.

Due to an old version of cloud-init that is bundled with upstream Stretch, we build custom base images (with modern cloud-init) for Stretch. Those are built on with bootstrap-vz on cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs.

== Building Pre-puppetized Images with wmcs-image-create ==

Our cloud should work with any upstream Debian base image that includes cloud-init version later than 18.4. The Debian project provides ready-made [https://cloud.debian.org/images/cloud/ images] prepared for use with OpenStack.

{{notice|To use the --image-url option with wmcs-image-create you must first enable the http proxy by running [[HTTP_proxy | set_proxy]].}}

For faster startup and better storage performance, we typically use a pre-puppetized snapshot based on an upstream image. The wmcs-image-create script will automatically generate and install a pre-puppetized image. This script is invoked with a url pointing to the desired upstream image to use as a starting point. It boots a temporary VM using that image, puppetized that VM, and creates a new image based on that puppetized VM.

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
# wmcs-image-create --image-url https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz --new-image-name &quot;debian-12 latest&quot; --project-owner &quot;testlabs&quot;
INFO:wmcs-image-create:Downloading upstream image...
INFO:wmcs-image-create:Running command:
    ('wget', 'https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz', '-O', PosixPath('/usr/local/sbin/wmcs-image-createax30c5bn/upstreamimage'))
    options: {}
--2025-05-29 17:39:18--  https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz
Resolving webproxy (webproxy)... 2620:0:861:3:208:80:154:74, 208.80.154.74
Connecting to webproxy (webproxy)|2620:0:861:3:208:80:154:74|:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz [following]
--2025-05-29 17:39:18--  https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.tar.xz
Connecting to webproxy (webproxy)|2620:0:861:3:208:80:154:74|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 215846020 (206M) [application/x-xz]
Saving to: ‘/usr/local/sbin/wmcs-image-createax30c5bn/upstreamimage’

/usr/local/sbin/wmcs-image-createax30c5bn/ 100%[======================================================================================&gt;] 205.85M  20.6MB/s    in 11s     

2025-05-29 17:39:30 (18.2 MB/s) - ‘/usr/local/sbin/wmcs-image-createax30c5bn/upstreamimage’ saved [215846020/215846020]

INFO:wmcs-image-create:Untarring and looking for disk.raw
INFO:wmcs-image-create:Running command:
    ('tar', 'xvf', PosixPath('/usr/local/sbin/wmcs-image-createax30c5bn/upstreamimage'))
    options: {'cwd': PosixPath('/usr/local/sbin/wmcs-image-createax30c5bn')}
INFO:wmcs-image-create:Creating a VM from the image and downloading a snapshot.
newimageid: 1b75f625-675f-453a-85c7-ec035ac825c1
INFO:wmcs-image-create:Launching a vm with the new image
INFO:wmcs-image-create:Created temporary VM 549f4fb3-c5eb-45b5-9cdd-6ffac3c02043
INFO:wmcs-image-create:Sleeping 5 minutes while we wait for the VM to start up...
INFO:wmcs-image-create:VM seems to be up!
INFO:wmcs-image-create:Waiting one minutes for VM to puppetize
INFO:wmcs-image-create:Waiting one minutes for VM to puppetize
INFO:wmcs-image-create:Stopping the VM
INFO:wmcs-image-create:Taking a snapshot of the stopped instance
INFO:wmcs-image-create:snapshot abb6de5f-26cf-4902-be44-b57ecd60af55 begun
INFO:wmcs-image-create:Grabbing handle to snapshot data
INFO:wmcs-image-create:Downloading snapshot to /usr/local/sbin/wmcs-image-createax30c5bn/snapshot.img
INFO:wmcs-image-create:Making snapshot file sparse
INFO:wmcs-image-create:Running command:
    ('cp', '--sparse=always', PosixPath('/usr/local/sbin/wmcs-image-createax30c5bn/snapshot.img'), PosixPath('/usr/local/sbin/wmcs-image-createax30c5bn/snapshot.img.sparse'))
    options: {}
INFO:wmcs-image-create:Creating final image
newimageid: 8352a8eb-4e12-4a01-adcd-a0b651557410
INFO:wmcs-image-create:Setting image ownership and visibility
INFO:wmcs-image-create:Cleaning up (removing instances, snapshots and temporary images)
INFO:wmcs-image-create:Cleaning up intermediate VM
INFO:wmcs-image-create:Cleaning up VM snapshot
INFO:wmcs-image-create:Cleaning up upstream image
INFO:wmcs-image-create:Finished creating new image: 8352a8eb-4e12-4a01-adcd-a0b651557410
&lt;/syntaxhighlight&gt;

The total run should take 10-15 minutes. Once the new image has been tested it needs to be renamed and marked public so that it's available to all projects:

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
# note IDs of outgoing and incoming image
root@cloudcontrol1003:~#  openstack image list
# Rename and deactivate the former image
root@cloudcontrol1003:~#  openstack image set --name &quot;debian-12.0-bookworm (replaced yyyy-mm-dd)&quot; --deactivate &lt;former active image id&gt;
# Publish the new image
root@cloudcontrol1003:~#  openstack image set --name &quot;debian-12.0-bookworm&quot; --public &lt;new image build id&gt;
&lt;/syntaxhighlight&gt;

== Legacy Build Process for Debian Stretch ==

=== Building with bootstrap-vz ===

* Login to &lt;code&gt;cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs&lt;/code&gt;
* Build and convert image

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sudo su -
cd /target
bootstrap-vz --pause-on-error  /etc/bootstrap-vz/manifests/labs-stretch.manifest.yaml
qemu-img convert -f raw -O qcow2 debian-stretch-amd64-$(date &quot;+%Y%m%d&quot;).raw debian-stretch-amd64-$(date &quot;+%Y%m%d&quot;).qcow2
&lt;/syntaxhighlight&gt;

==== Local bootstrap-vz for stretch ====

The version of bootstrap-vz built for Stretch doesn't support setting mount options in the manifest. Since we want 'discard' set, the local install of bootstrapvz on cloud-bootstrapvz-stretch.openstack.eqiad1.wikimedia.cloud has a hack in place to make this setting, in bootstrapvz/common/tasks/filesystem.py:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
190c190
&lt;             mount_opts = ['defaults']
---
&gt;             mount_opts = ['discard', 'defaults']
&lt;/syntaxhighlight&gt;

=== How To Test ===

You can boot an image locally for testing, like this:

&lt;code&gt;
sudo qemu-system-x86_64 -nographic -serial mon:stdio -enable-kvm image_name.raw
&lt;/code&gt;

If the command above does not work, you try can try the following command (beware boot logs will be supressed):

&lt;code&gt;
qemu-system-x86_64 image_name.raw --curses
&lt;/code&gt;

Having a working login account for test purposes is left as an exercise to the reader. bootstrap-vz creates one by default (login:root / passwd:test) but our config wisely disables it.

=== How To Deploy ===

Images are deployed to the OpenStack Glance service.

* Copy the .qcow2 file to /tmp on the &lt;code&gt;cloudcontrol1003.wikimedia.org&lt;/code&gt; server

Since the file has to cross the Cloud VPS / Production boundary, you can copy it from the builder server to your laptop (using your Cloud Services root key) and then from your laptop to cloudcontrol1003 (using your production key):

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
rsync --progress -v -e ssh root@cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs:/target/debian-stretch-amd64-$(date &quot;+%Y%m%d&quot;).qcow2 .
rsync --progress -v -e ssh debian-stretch-amd64-$(date &quot;+%Y%m%d&quot;).qcow2 cloudcontrol1003.wikimedia.org:/tmp/
&lt;/syntaxhighlight&gt;

Alternatively, you can open a temporary HTTP server to make this transfer:

&lt;syntaxhighlight lang=&quot;shell-session&gt;
cloud-bootstrapvz-stretch:~ $ cd /target
cloud-bootstrapvz-stretch:~ $ python3 -m http.server 80

cloudcontrol1003:/tmp$ wget http://185.15.56.45/debian-stretch-amd64-$(date &quot;+%Y%m%d&quot;).qcow2
&lt;/syntaxhighlight&gt;

* Login to &lt;code&gt;cloudcontrol1003.wikimedia.org&lt;/code&gt;
* Convert the image from qcow2 to raw/sparse.  This is a new step necessary now that images are stored using ceph/rbd: 
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
qemu-img convert -f qcow2 -O raw  ./&lt;filename&gt;.qcow2 ./&lt;filename&gt;.raw.notsparse
cp --sparse=always ./&lt;filename&gt;.raw.notsparse  ./&lt;filename&gt;.raw
&lt;/syntaxhighlight&gt;
* Create new image in Glance:

{{warning |  Only rename the existing images AFTER you upload the new images. We have [[Nova_Resource:Admin-monitoring|monitoring]] in place that depends on the exact image names. Remember to update it as well (and wait for Puppet to run and update fullstackd). }}

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sudo su -
source ~/novaenv.sh 
cd /tmp
openstack image create --file debian-stretch-amd64-$(date &quot;+%Y%m%d&quot;).raw --disk-format &quot;raw&quot; --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi --container-format &quot;ovf&quot; --public &quot;debian-9.6-stretch&quot;
&lt;/syntaxhighlight&gt;

* Test new image by booting a new VM with it (if the image is faulty, remember to delete the test VM and the faulty image)
* Update fullstackd to use this new image (see T218314 for an example).
* Get a list of existing images
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
openstack image list
&lt;/syntaxhighlight&gt;
* Append &quot;deprecated&quot; to old images and remove properties (only if new image is working as expected)

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
openstack image set --name &quot;debian-9.5-stretch (replaced &lt;date&gt;)&quot; &lt;old-image-id&gt;
nova image-meta &lt;old-image-id&gt; delete default
nova image-meta &lt;old-image-id&gt; delete show
&lt;/syntaxhighlight&gt;

Passing &lt;code&gt;--purge-props&lt;/code&gt; to &lt;code&gt;openstack image set&lt;/code&gt; should be enough to clear all properties but it's currently not available in our OpenStack version. The &lt;code&gt;nova image-meta&lt;/code&gt; commands serve the same purpose but you have to delete each property individually. This should be reviewed when OpenStack is upgraded.

* Make the new image the default for new instances

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
openstack image set --name &quot;debian-9.6-stretch&quot; &lt;new-image-id&gt; --property show=true --property default=true
&lt;/syntaxhighlight&gt;

Notice in the above &lt;code&gt;glance image-update&lt;/code&gt; commands the use of properties. If &lt;code&gt;default=true&lt;/code&gt; the image will be the default image selected in the instance creation interface. Purging properties removes the 'default' state.

== Deleting Unused Images ==

{{Warning | '''WARNING!''' You should never delete old images from Glance as long as there are VMs based on those base images.}}

Don't delete glance images as long as there are VMs runnning them:
* Resizing a VM requires access to the base image it was launched with. See how [[Portal:Cloud_VPS/Admin/Resizing_instances|resizing instances]] works to understand it.
* We should always be able to identify the OS an instance is running based on the image name.

You can locate unused images with the wmcs-imageusage script:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@cloudcontrol1003:~# wmcs-imageusage 
 -- unknown image 249c601a-42c8-4ba7-ba7a-878fba4ef799
 -- unknown image 68e61634-d599-4c6b-94da-10e6c7d36573
ea564b1c-9e5d-45a9-9d53-2fe023344e3f: debian-9.13-stretch (deprecated 2021-01-22), 0
a64f590c-3ed7-43e1-a592-b44d86f10641: debian-10.0-buster (deprecated 2019-07-30), 0
86105151-0d2e-498d-95b3-da712f19c7e2: debian-9.0-stretch (deprecated 2017-08-03), 0
559facb6-532f-47b0-9fa2-f0c0207c00c9: debian-9.13-stretch, 1
baa92f56-8aca-4855-a0f6-47d94e8a2167: debian-9.11-stretch (deprecated 2019-12-18), 1
6255ff81-db78-4521-b770-076fe16d365f: debian-9.11-stretch (deprecated 2019-12-15), 1
d5d88ba0-0def-4cfb-8c7c-cae8946722d8: debian-8.0-jessie (deprecated 2015-06-13), 1
160313be-b1b1-4b62-8906-bee26f25f6dc: debian-9.13-stretch (deprecated 2021-03-11), 2
f345f03d-a601-48cf-951b-ad1b4ad8f551: debian-9.13-stretch (deprecated 2021-01-20), 2
5cd88504-5e2b-4f63-b18e-2ec935c914dd: debian-10.0-buster-prerelease, 2
b7c4fc02-433f-4b49-97c2-b2492012b742: debian-8.2-jessie (deprecated 2016-02-16), 2
d633c111-00d1-4e72-bb3b-c2bd60518f4a: debian-10.0-buster, 3
fc6fb78b-4515-4dcc-8254-591b9fe01762: debian-10.0-buster (deprecated 2019-12-18), 3
e27bab24-4a17-47e5-bb89-b0c7526dea20: debian-9.2-stretch (deprecated 2017-12-13), 3
b7616101-3b9e-4948-98c7-25582234e788: debian-9.0-stretch (deprecated 2017-09-27), 3
0e33dcf3-37fe-416c-bc00-3abd84dda054: debian-9.0-stretch (deprecated 2017-07-19), 3
ad7bee1a-a890-4fed-b851-6c02138683b0: debian-9.13-stretch (deprecated 2021-03-01), 4
374ca3c6-6c4b-4e03-9f31-52e0d44aae0c: debian-10.0-buster (deprecated 2019-07-29), 4
d620d77c-c023-41ae-944c-2f10063bfc77: debian-9.6-stretch (deprecated 2019-02-14), 4
bb37bd5c-3cc5-4ee6-82f8-473e9568ff44: debian-9.3-stretch (deprecated 2018-01-10), 4
13b0883f-2ce6-467a-ba18-17484413faaa: debian-9.8-stretch (deprecated 2019-04-02), 7
e02770ae-b45f-4776-a852-d9a13217611e: debian-9.1-stretch (deprecated  2017-11-16), 7
325ce1c8-2f95-4e98-8088-f7b46e7a6bb5: debian-9.9-stretch (deprecated 2019-09-21), 8
e3716d55-5278-4f9b-a30f-41db9cb23ef8: debian-9.7-stretch (deprecated 2019-03-14), 10
b7274e93-30f4-4567-88aa-46223c59107e: debian-9.4-stretch (deprecated 2018-08-01), 12
73ccc348-f69d-4cd5-9f09-d83ea222fa02: debian-9.11-stretch (deprecated 2019-11-05), 13
64351116-a53e-4a62-8866-5f0058d89c2b: debian-10.0-buster (deprecated 2021-03-01), 18
6e6d743a-64f0-4cd3-b1b9-327bbf57e03b: debian-9.3-stretch (deprecated 2018-06-05), 19
25eeb420-1977-45f2-bbd9-fb65f48c0947: debian-9.11-stretch (deprecated 2020-10-17), 21
10783e59-b30e-4426-b509-2fbef7d3103c: debian-9.8-stretch (deprecated 2019-07-25), 30
e971dc0f-3b5c-4cd2-ab8b-02faf403c136: debian-10.0-buster (deprecated 2021-03-24), 46
c6273cce-9b8b-4364-9f1f-7bf58436994f: debian-9.5-stretch (deprecated 2018-11-22), 49
b6b58ba2-8656-49b4-af13-d0530ac05365: debian-10.0-buster (deprecated 2019-12-15), 71
7c6371d1-8411-48c7-bf73-2ef6d6ff2a15: debian-9.6-stretch (deprecated 2019-01-22), 83
6b67c8a1-6356-464d-a885-0576d7263e51: debian-10.0-buster (deprecated 2021-02-22), 92
031d2d76-8368-4066-a502-d28107d0195e: debian-10.0-buster (deprecated 2020-10-16), 227
&lt;/syntaxhighlight&gt;

Images are cheap, though, so generally image cleanup is unwarranted. Typically we only clean up images that were explicitly single-use (e.g. created to troubleshoot the image creation logic) or images for distributions that we no longer support and which no longer exist within our cloud (e.g. Debian Jessie.)

Deleting an image from glance is a single command:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@cloudcontrol1003:~# openstack image delete &lt;imageid&gt;
&lt;/syntaxhighlight&gt;

If this fails with a warning that the image is used by VMs, stop now!  Often, though, this command will fail due to out-of-band snapshots (e.g. those created by the image backup service.)

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@cloudcontrol1003:~# openstack image delete 11d52ebe-e4a0-45af-99ea-8f286ed55696
Failed to delete image with name or ID '11d52ebe-e4a0-45af-99ea-8f286ed55696': HTTP 409 Conflict: The image cannot be deleted because it has snapshot(s).
Failed to delete 1 of 1 images.
&lt;/syntaxhighlight&gt;

To delete all snapshots for a given image, use 'rbd snap purge':

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@cloudcontrol1003:~# rbd snap purge eqiad1-glance-images/11d52ebe-e4a0-45af-99ea-8f286ed55696
Removing all snapshots: 100% complete...done.
&lt;/syntaxhighlight&gt;

Once snapshots are removed the image can be removed with 'openstack image delete'.

== {{anchor|Private Images}} Restricted images ==
It's possible to restrict an image to a single project or set of projects.  This is useful when e.g. gradually deprecating use of an OS.

First, mark the image as &quot;shared&quot; and restrict it to the desired project:
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
$ ssh cloudcontrol1004.wikimedia.org
$ sudo wmcs-openstack image set --property visibility=shared --project $PROJECT_ID $IMAGE_UUID
$ sudo wmcs-openstack image set --activate $IMAGE_UUID
&lt;/syntaxhighlight&gt;

Then, allow the 'observer' user to access it so that [[toolforge:openstack-browser|OpenStack browser]] can display something more useful than &quot;UNKNOWN&quot; for instances created from it:
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
$ sudo su -
$ source ~/novaenv.sh
$ glance member-create $IMAGE_UUID observer
$ glance member-update $IMAGE_UUID observer accepted
&lt;/syntaxhighlight&gt;
{{Fixme|1=can this be done with &lt;code&gt;openstack&lt;/code&gt; instead of &lt;code&gt;glance&lt;/code&gt;?}}

== How To Deactivate Obsolete Images ==

We used to use the 'show=true' property to ensure that an image appeared on Wikitech. Now instead we use the image state, where only images with state=active appear in the GUI (both on wikitech and in Horizon.) To deactivate your obsolete image:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
$ sudo wmcs-openstack image set --deactivate &lt;image-id&gt;
&lt;/syntaxhighlight&gt;

If you need to reactive it for some reason:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
$ sudo wmcs-openstack image set --activate &lt;image-id&gt;
&lt;/syntaxhighlight&gt;

Please note that we usually just &quot;deprecate&quot; images by changing their names. Deactivating an image is a more extreme step to be used when you do not want any users to have access to it.

== Internals ==

=== bootstrap-vz configuration files ===

Bootstrap-vz uses source files from /etc/bootstrap-vz. These files are puppetized, so you'll want to disable Puppet if you change them.

{| class=&quot;wikitable&quot;
|-
! OS !! Configuration File
|-
| Debian Stretch || /etc/bootstrap-vz/manifests/labs-stretch.manifest.yaml
|-
| Debian Jessie || /etc/bootstrap-vz/manifests/labs-jessie.manifest.yaml
|}


Bootstrap-vz also uses several source files that are standard local config files on the build host. For a complete list of these files, look at the 'file_copy:' section in /etc/bootstrap-vz/manifests/labs-{stretch,jessie}.manifest.yaml

=== First Boot ===

The first boot of the VM image is a key moment in the setup of an instance in Cloud VPS.

This is usually done by means of the [[phab:source/operations-puppet/browse/production/modules/labs_bootstrapvz/files/firstboot.sh | /root/firstboot.sh]] script which is called by means of [[phab:source/operations-puppet/browse/production/modules/labs_bootstrapvz/files/firstbootrc | /etc/rc.local]].

The script will do:

* some LVM configuration
* run DHCP request for configuration
* name/domain resolution to autoconfigure the VM
* initial puppet autoconfiguration (cert request, etc)
* initial configuration of nscd/nslcd
* initial apt updates
* NFS mounts if required
* final puppet run to fetch all remaining configuration (ssh keys, packages, etc)

Until the last point, the instance may have limited connectivity or usability.

== Troubleshooting ==

{{note | For general Cloud VPS troubleshooting, please read [[Portal:Cloud_VPS/Admin/Troubleshooting | the operational troubleshooting documents]].}}
This troubleshooting section is specific for VM images (in glance) and generally only usefull when dealing with new VM images.

=== Common Issues ===

Common issues when dealing with VM images. These problems may vary from [[Portal:Cloud_VPS/Admin/Deployments | deployment]] to deployment, but they could be common.

* Image does not have the puppet master CA, so it fails to fetch catalog (see [[phab:T181523]])
* Image does not have the puppet master CRL, so it fails to fetch catalog (see [[phab:T181523]])
* Image doesn't correctly resolve the hostname/domain name (so it fails to fetch its own puppet catalog)

=== How To Inspect Disk Contents ===

If you want to explore and edit the disk image of a ''live instance'', read the docs at [[Portal:Cloud_VPS/Admin/Troubleshooting#Mounting_an_instance_disk | Cloud VPS troubleshooting, mounting an instance disk]].

=== How to fix VM disk corruption ===

Please read [[Portal:Cloud_VPS/Admin/Troubleshooting#Fix_VM_disk_corruption_(fsck) | Cloud VPS troubleshooting, fixing VM disk corruption (fsck)]].

== Building an image from an existing VM ==

Nova can create a new base image out of an existing VM. This isn't something that we've done much but it's potentially useful to e.g. rapidly create a fleet of worker nodes from an original template.

* Stop the source VM
 openstack server stop &lt;instance-id&gt;
* Create the new image
 openstack server image create &lt;instance-id&gt; --name &lt;new image name&gt;
* Wait for the image state to change from 'queued' to 'saving' to 'active'
 openstack image show &lt;new image id&gt;
* Set project ownership
 openstack image set --shared &lt;new image id&gt;
 openstack image add project &lt;new image id&gt; &lt;project&gt;

== See also ==

* [[Portal:Cloud_VPS/Admin/Maintenance]]: Information on Cloud VPS maintenance tasks</text>
      <sha1>8l0g17zimu0na7385gz1mjvkppx129p</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2435509</id>
      <parentid>2435484</parentid>
      <timestamp>2026-07-12T00:01:41Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2435509</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="110" sha1="bw8ff6xqn6ca4ncyu1uio0ot8c679n0" xml:space="preserve">{{/Header}}
== Today (2026-07-12) ==
== Yesterday (2026-07-11) ==
== Last seven days ==


[[Category:MariaDB]]</text>
      <sha1>bw8ff6xqn6ca4ncyu1uio0ot8c679n0</sha1>
    </revision>
  </page>
  <page>
    <title>Talk:Anycast</title>
    <ns>1</ns>
    <id>452570</id>
    <revision>
      <id>2435519</id>
      <parentid>2072300</parentid>
      <timestamp>2026-07-12T01:27:34Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>/* Removal of Routing configuration section */ Reply</comment>
      <origin>2435519</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="718" sha1="t95tpm77cn8ukxd9squfhivq8v9jmuy" xml:space="preserve">== Removal of Routing configuration section ==

I propose removing that section as copy/pasting sections of code is destined to drift apart. If one wants to view the configuration, a link to the code could be put in here. [[User:BCornwall|BCornwall]] ([[User talk:BCornwall|talk]]) 20:24, 27 April 2023 (UTC)

:@[[User:BCornwall|BCornwall]] Linking the code sounds good. Nobody has objected, so I think this should be done. 
:I tried searching for the relevant file ([https://codesearch.wmcloud.org/search/?q=Anycast&amp;files=&amp;excludeFiles=&amp;repos= codesearch]), but I couldn't easily determine if it was one of those, or stored elsewhere. [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]]) 01:27, 12 July 2026 (UTC)</text>
      <sha1>t95tpm77cn8ukxd9squfhivq8v9jmuy</sha1>
    </revision>
    <revision>
      <id>2435520</id>
      <parentid>2435519</parentid>
      <timestamp>2026-07-12T01:29:19Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>/* Removal of Routing configuration section */ Reply</comment>
      <origin>2435520</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="881" sha1="eljrlx9oimbt9oq2p1st8bf3fgw1cjl" xml:space="preserve">== Removal of Routing configuration section ==

I propose removing that section as copy/pasting sections of code is destined to drift apart. If one wants to view the configuration, a link to the code could be put in here. [[User:BCornwall|BCornwall]] ([[User talk:BCornwall|talk]]) 20:24, 27 April 2023 (UTC)

:@[[User:BCornwall|BCornwall]] Linking the code sounds good. Nobody has objected, so I think this should be done. 
:I tried searching for the relevant file ([https://codesearch.wmcloud.org/search/?q=Anycast&amp;files=&amp;excludeFiles=&amp;repos= codesearch]), but I couldn't easily determine if it was one of those, or stored elsewhere. [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]]) 01:27, 12 July 2026 (UTC)
::Re-ping @[[User:BCornwall-WMF|User:BCornwall-WMF]] with current username. :-) [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]]) 01:29, 12 July 2026 (UTC)</text>
      <sha1>eljrlx9oimbt9oq2p1st8bf3fgw1cjl</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:Gitlab-account-approval/Log</title>
    <ns>116</ns>
    <id>453906</id>
    <revision>
      <id>2435490</id>
      <parentid>2435431</parentid>
      <timestamp>2026-07-11T12:18:14Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>pratyushsawan was rejected.</comment>
      <origin>2435490</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="60607" sha1="d1bi5ohpc8xm7rpzfltreyaktwmfdci" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-07-11 ===
* 12:18 &quot;pratyushsawan&quot; was rejected (pending since 2026-04-11T12:16:41.671Z).

=== 2026-07-10 ===
* 14:42 [[gitlab:akaza24|@akaza24]] was approved.
* 12:57 [[gitlab:kormisk|@kormisk]] was approved.

=== 2026-07-07 ===
* 10:36 [[gitlab:olafjanssen|@olafjanssen]] was approved.
* 06:57 &quot;elisapoly-99&quot; was rejected (pending since 2026-04-07T06:55:52.662Z).

=== 2026-07-06 ===
* 11:57 &quot;ma3rouf&quot; was rejected (pending since 2026-04-06T11:56:30.978Z).

=== 2026-07-02 ===
* 15:48 [[gitlab:tekneos|@tekneos]] was approved.

=== 2026-07-01 ===
* 15:12 [[gitlab:mugurolevy|@mugurolevy]] was approved.
* 14:15 [[gitlab:vadymts1|@vadymts1]] was approved.
* 09:57 &quot;mugurolevy&quot; was rejected (pending since 2026-04-01T09:55:19.175Z).

=== 2026-06-30 ===
* 14:27 &quot;shivangisharma&quot; was rejected (pending since 2026-03-31T14:26:44.932Z).

=== 2026-06-29 ===
* 19:03 [[gitlab:thisismattmiller|@thisismattmiller]] was approved.

=== 2026-06-28 ===
* 14:51 &quot;nkwenuinadine&quot; was rejected (pending since 2026-03-29T14:48:32.735Z).
* 14:03 &quot;vaishnavikumbhar&quot; was rejected (pending since 2026-03-29T14:01:30.604Z).
* 13:03 &quot;stepmay&quot; was rejected (pending since 2026-03-29T13:01:59.905Z).
* 06:51 &quot;swallroth&quot; was rejected (pending since 2026-03-29T06:49:54.838Z).

=== 2026-06-26 ===
* 09:39 [[gitlab:lakshita28|@lakshita28]] was approved.
* 07:30 [[gitlab:reeti|@reeti]] was approved.
* 07:30 [[gitlab:anushka10patel|@anushka10patel]] was approved.
* 07:30 &quot;samsaesque&quot; was rejected (pending since 2026-03-27T07:29:57.279Z).
* 05:51 [[gitlab:arpithhhaaa|@arpithhhaaa]] was approved.
* 05:51 [[gitlab:govindlaltl|@govindlaltl]] was approved.

=== 2026-06-25 ===
* 16:09 [[gitlab:sakuraemad|@sakuraemad]] was approved.
* 07:00 &quot;kdh8219&quot; was rejected (pending since 2026-03-26T06:58:05.415Z).

=== 2026-06-24 ===
* 11:54 [[gitlab:sanskardubeydev|@sanskardubeydev]] was approved.
* 10:09 &quot;tanmay789q&quot; was rejected (pending since 2026-03-25T10:07:54.602Z).

=== 2026-06-22 ===
* 19:57 [[gitlab:gouvernathor|@gouvernathor]] was approved.
* 16:45 [[gitlab:lucasbelo|@lucasbelo]] was approved.
* 07:15 &quot;jason2000-cpu&quot; was rejected (pending since 2026-03-23T07:14:09.184Z).

=== 2026-06-21 ===
* 13:18 [[gitlab:egonw|@egonw]] was approved.

=== 2026-06-20 ===
* 10:21 [[gitlab:tways2017|@tways2017]] was approved.

=== 2026-06-19 ===
* 16:06 &quot;wilsonwang2026&quot; was rejected (pending since 2026-03-20T16:06:05.511Z).
* 04:12 [[gitlab:claudio|@claudio]] was approved.

=== 2026-06-18 ===
* 14:21 &quot;royiswariii&quot; was rejected (pending since 2026-03-19T14:19:16.896Z).
* 13:06 [[gitlab:laurabarluzzi|@laurabarluzzi]] was approved.

=== 2026-06-17 ===
* 11:24 &quot;adinathq8x&quot; was rejected (pending since 2026-03-18T11:22:50.098Z).
* 09:45 &quot;nathanveritas&quot; was rejected (pending since 2026-03-18T09:43:51.645Z).

=== 2026-06-15 ===
* 22:39 [[gitlab:mohammadhijjawi|@mohammadhijjawi]] was approved.
* 14:24 &quot;enlisar&quot; was rejected (pending since 2026-03-16T14:23:00.109Z).
* 14:06 &quot;ayaan&quot; was rejected (pending since 2026-03-16T14:03:31.071Z).
* 10:54 &quot;kwametech&quot; was rejected (pending since 2026-03-16T10:54:11.083Z).

=== 2026-06-14 ===
* 17:45 [[gitlab:surajseth520|@surajseth520]] was approved.
* 07:24 &quot;malahimhaseeb&quot; was rejected (pending since 2026-03-15T07:21:57.748Z).

=== 2026-06-11 ===
* 11:48 [[gitlab:cadddr|@cadddr]] was approved.
* 11:18 &quot;wikipiggy&quot; was rejected (pending since 2026-03-12T11:16:09.335Z).
* 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved.

=== 2026-06-10 ===
* 07:03 [[gitlab:dmiranda|@dmiranda]] was approved.

=== 2026-06-09 ===
* 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved.
* 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved.
* 12:51 [[gitlab:ekrem|@ekrem]] was approved.

=== 2026-06-08 ===
* 17:48 &quot;jmprax&quot; was rejected (pending since 2026-03-09T17:46:38.807Z).
* 16:15 [[gitlab:ahonc|@ahonc]] was approved.
* 12:57 [[gitlab:rainmonger|@rainmonger]] was approved.

=== 2026-06-07 ===
* 23:03 &quot;shadowthewuff&quot; was rejected (pending since 2026-03-08T23:00:53.442Z).
* 11:45 &quot;wiki-pavan&quot; was rejected (pending since 2026-03-08T11:45:11.116Z).
* 02:39 [[gitlab:launchpad|@launchpad]] was approved.

=== 2026-06-06 ===
* 14:54 &quot;unicord&quot; was rejected (pending since 2026-03-07T14:52:04.992Z).
* 12:48 &quot;chien&quot; was rejected (pending since 2026-03-07T12:48:11.669Z).

=== 2026-06-04 ===
* 14:33 &quot;only-vikas&quot; was rejected (pending since 2026-03-05T14:32:09.186Z).

=== 2026-06-03 ===
* 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved.

=== 2026-06-02 ===
* 21:21 &quot;mgagat&quot; was rejected (pending since 2026-03-03T21:18:37.223Z).
* 13:57 &quot;prasunaenumarthy&quot; was rejected (pending since 2026-03-03T13:57:14.847Z).
* 05:48 [[gitlab:tmoney|@tmoney]] was approved.

=== 2026-06-01 ===
* 14:57 &quot;vikram2101&quot; was rejected (pending since 2026-03-02T14:54:26.550Z).
* 12:03 &quot;watshell&quot; was rejected (pending since 2026-03-02T12:03:09.329Z).

=== 2026-05-29 ===
* 12:48 &quot;mounikapotladurthi&quot; was rejected (pending since 2026-02-27T12:45:38.609Z).

=== 2026-05-27 ===
* 20:00 &quot;vinitha&quot; was rejected (pending since 2026-02-25T19:58:43.524Z).
* 16:30 &quot;codeurluce&quot; was rejected (pending since 2026-02-25T16:28:53.973Z).
* 14:33 [[gitlab:thilio|@thilio]] was approved.

=== 2026-05-26 ===
* 12:09 &quot;charisad&quot; was rejected (pending since 2026-02-24T12:07:21.881Z).

=== 2026-05-25 ===
* 22:54 &quot;ddshelto&quot; was rejected (pending since 2026-02-23T22:52:44.427Z).
* 19:51 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).
* 19:48 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).

=== 2026-05-24 ===
* 18:45 &quot;jiyagupta-cs&quot; was rejected (pending since 2026-02-22T18:43:33.176Z).

=== 2026-05-23 ===
* 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved.
* 04:21 [[gitlab:staraction|@staraction]] was approved.

=== 2026-05-22 ===
* 19:03 &quot;i-horich&quot; was rejected (pending since 2026-02-20T19:00:43.519Z).
* 01:48 &quot;50323233&quot; was rejected (pending since 2026-02-20T01:48:05.555Z).

=== 2026-05-21 ===
* 18:51 &quot;kartikeyg0104&quot; was rejected (pending since 2026-02-19T18:48:39.707Z).
* 16:27 [[gitlab:renovatebot|@renovatebot]] was approved.
* 16:06 [[gitlab:gkm563|@gkm563]] was approved.

=== 2026-05-20 ===
* 01:21 &quot;beedellrokejulianlockhart&quot; was rejected (pending since 2026-02-18T01:19:13.284Z).

=== 2026-05-18 ===
* 23:18 &quot;wladek92&quot; was rejected (pending since 2026-02-16T23:16:22.939Z).
* 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved.

=== 2026-05-14 ===
* 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved.

=== 2026-05-13 ===
* 10:51 &quot;ssssaaaa&quot; was rejected (pending since 2026-02-11T10:50:36.975Z).

=== 2026-05-12 ===
* 18:06 [[gitlab:psubhashish|@psubhashish]] was approved.
* 08:12 &quot;khan&quot; was rejected (pending since 2026-02-10T08:11:48.776Z).
* 04:27 &quot;galaxysh&quot; was rejected (pending since 2026-02-10T04:24:59.440Z).

=== 2026-05-11 ===
* 12:18 &quot;peterxy12&quot; was rejected (pending since 2026-02-09T12:18:01.982Z).

=== 2026-05-10 ===
* 11:09 &quot;yalihupokn&quot; was rejected (pending since 2026-02-08T11:06:51.336Z).
* 05:12 &quot;wobadha&quot; was rejected (pending since 2026-02-08T05:11:00.569Z).

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>d1bi5ohpc8xm7rpzfltreyaktwmfdci</sha1>
    </revision>
    <revision>
      <id>2435494</id>
      <parentid>2435490</parentid>
      <timestamp>2026-07-11T14:48:13Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>bigcereal42 was rejected.</comment>
      <origin>2435494</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="60684" sha1="ertv87exmjsoqti0wlnhpml7l78cstq" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-07-11 ===
* 14:48 &quot;bigcereal42&quot; was rejected (pending since 2026-04-11T14:47:40.321Z).
* 12:18 &quot;pratyushsawan&quot; was rejected (pending since 2026-04-11T12:16:41.671Z).

=== 2026-07-10 ===
* 14:42 [[gitlab:akaza24|@akaza24]] was approved.
* 12:57 [[gitlab:kormisk|@kormisk]] was approved.

=== 2026-07-07 ===
* 10:36 [[gitlab:olafjanssen|@olafjanssen]] was approved.
* 06:57 &quot;elisapoly-99&quot; was rejected (pending since 2026-04-07T06:55:52.662Z).

=== 2026-07-06 ===
* 11:57 &quot;ma3rouf&quot; was rejected (pending since 2026-04-06T11:56:30.978Z).

=== 2026-07-02 ===
* 15:48 [[gitlab:tekneos|@tekneos]] was approved.

=== 2026-07-01 ===
* 15:12 [[gitlab:mugurolevy|@mugurolevy]] was approved.
* 14:15 [[gitlab:vadymts1|@vadymts1]] was approved.
* 09:57 &quot;mugurolevy&quot; was rejected (pending since 2026-04-01T09:55:19.175Z).

=== 2026-06-30 ===
* 14:27 &quot;shivangisharma&quot; was rejected (pending since 2026-03-31T14:26:44.932Z).

=== 2026-06-29 ===
* 19:03 [[gitlab:thisismattmiller|@thisismattmiller]] was approved.

=== 2026-06-28 ===
* 14:51 &quot;nkwenuinadine&quot; was rejected (pending since 2026-03-29T14:48:32.735Z).
* 14:03 &quot;vaishnavikumbhar&quot; was rejected (pending since 2026-03-29T14:01:30.604Z).
* 13:03 &quot;stepmay&quot; was rejected (pending since 2026-03-29T13:01:59.905Z).
* 06:51 &quot;swallroth&quot; was rejected (pending since 2026-03-29T06:49:54.838Z).

=== 2026-06-26 ===
* 09:39 [[gitlab:lakshita28|@lakshita28]] was approved.
* 07:30 [[gitlab:reeti|@reeti]] was approved.
* 07:30 [[gitlab:anushka10patel|@anushka10patel]] was approved.
* 07:30 &quot;samsaesque&quot; was rejected (pending since 2026-03-27T07:29:57.279Z).
* 05:51 [[gitlab:arpithhhaaa|@arpithhhaaa]] was approved.
* 05:51 [[gitlab:govindlaltl|@govindlaltl]] was approved.

=== 2026-06-25 ===
* 16:09 [[gitlab:sakuraemad|@sakuraemad]] was approved.
* 07:00 &quot;kdh8219&quot; was rejected (pending since 2026-03-26T06:58:05.415Z).

=== 2026-06-24 ===
* 11:54 [[gitlab:sanskardubeydev|@sanskardubeydev]] was approved.
* 10:09 &quot;tanmay789q&quot; was rejected (pending since 2026-03-25T10:07:54.602Z).

=== 2026-06-22 ===
* 19:57 [[gitlab:gouvernathor|@gouvernathor]] was approved.
* 16:45 [[gitlab:lucasbelo|@lucasbelo]] was approved.
* 07:15 &quot;jason2000-cpu&quot; was rejected (pending since 2026-03-23T07:14:09.184Z).

=== 2026-06-21 ===
* 13:18 [[gitlab:egonw|@egonw]] was approved.

=== 2026-06-20 ===
* 10:21 [[gitlab:tways2017|@tways2017]] was approved.

=== 2026-06-19 ===
* 16:06 &quot;wilsonwang2026&quot; was rejected (pending since 2026-03-20T16:06:05.511Z).
* 04:12 [[gitlab:claudio|@claudio]] was approved.

=== 2026-06-18 ===
* 14:21 &quot;royiswariii&quot; was rejected (pending since 2026-03-19T14:19:16.896Z).
* 13:06 [[gitlab:laurabarluzzi|@laurabarluzzi]] was approved.

=== 2026-06-17 ===
* 11:24 &quot;adinathq8x&quot; was rejected (pending since 2026-03-18T11:22:50.098Z).
* 09:45 &quot;nathanveritas&quot; was rejected (pending since 2026-03-18T09:43:51.645Z).

=== 2026-06-15 ===
* 22:39 [[gitlab:mohammadhijjawi|@mohammadhijjawi]] was approved.
* 14:24 &quot;enlisar&quot; was rejected (pending since 2026-03-16T14:23:00.109Z).
* 14:06 &quot;ayaan&quot; was rejected (pending since 2026-03-16T14:03:31.071Z).
* 10:54 &quot;kwametech&quot; was rejected (pending since 2026-03-16T10:54:11.083Z).

=== 2026-06-14 ===
* 17:45 [[gitlab:surajseth520|@surajseth520]] was approved.
* 07:24 &quot;malahimhaseeb&quot; was rejected (pending since 2026-03-15T07:21:57.748Z).

=== 2026-06-11 ===
* 11:48 [[gitlab:cadddr|@cadddr]] was approved.
* 11:18 &quot;wikipiggy&quot; was rejected (pending since 2026-03-12T11:16:09.335Z).
* 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved.

=== 2026-06-10 ===
* 07:03 [[gitlab:dmiranda|@dmiranda]] was approved.

=== 2026-06-09 ===
* 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved.
* 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved.
* 12:51 [[gitlab:ekrem|@ekrem]] was approved.

=== 2026-06-08 ===
* 17:48 &quot;jmprax&quot; was rejected (pending since 2026-03-09T17:46:38.807Z).
* 16:15 [[gitlab:ahonc|@ahonc]] was approved.
* 12:57 [[gitlab:rainmonger|@rainmonger]] was approved.

=== 2026-06-07 ===
* 23:03 &quot;shadowthewuff&quot; was rejected (pending since 2026-03-08T23:00:53.442Z).
* 11:45 &quot;wiki-pavan&quot; was rejected (pending since 2026-03-08T11:45:11.116Z).
* 02:39 [[gitlab:launchpad|@launchpad]] was approved.

=== 2026-06-06 ===
* 14:54 &quot;unicord&quot; was rejected (pending since 2026-03-07T14:52:04.992Z).
* 12:48 &quot;chien&quot; was rejected (pending since 2026-03-07T12:48:11.669Z).

=== 2026-06-04 ===
* 14:33 &quot;only-vikas&quot; was rejected (pending since 2026-03-05T14:32:09.186Z).

=== 2026-06-03 ===
* 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved.

=== 2026-06-02 ===
* 21:21 &quot;mgagat&quot; was rejected (pending since 2026-03-03T21:18:37.223Z).
* 13:57 &quot;prasunaenumarthy&quot; was rejected (pending since 2026-03-03T13:57:14.847Z).
* 05:48 [[gitlab:tmoney|@tmoney]] was approved.

=== 2026-06-01 ===
* 14:57 &quot;vikram2101&quot; was rejected (pending since 2026-03-02T14:54:26.550Z).
* 12:03 &quot;watshell&quot; was rejected (pending since 2026-03-02T12:03:09.329Z).

=== 2026-05-29 ===
* 12:48 &quot;mounikapotladurthi&quot; was rejected (pending since 2026-02-27T12:45:38.609Z).

=== 2026-05-27 ===
* 20:00 &quot;vinitha&quot; was rejected (pending since 2026-02-25T19:58:43.524Z).
* 16:30 &quot;codeurluce&quot; was rejected (pending since 2026-02-25T16:28:53.973Z).
* 14:33 [[gitlab:thilio|@thilio]] was approved.

=== 2026-05-26 ===
* 12:09 &quot;charisad&quot; was rejected (pending since 2026-02-24T12:07:21.881Z).

=== 2026-05-25 ===
* 22:54 &quot;ddshelto&quot; was rejected (pending since 2026-02-23T22:52:44.427Z).
* 19:51 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).
* 19:48 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).

=== 2026-05-24 ===
* 18:45 &quot;jiyagupta-cs&quot; was rejected (pending since 2026-02-22T18:43:33.176Z).

=== 2026-05-23 ===
* 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved.
* 04:21 [[gitlab:staraction|@staraction]] was approved.

=== 2026-05-22 ===
* 19:03 &quot;i-horich&quot; was rejected (pending since 2026-02-20T19:00:43.519Z).
* 01:48 &quot;50323233&quot; was rejected (pending since 2026-02-20T01:48:05.555Z).

=== 2026-05-21 ===
* 18:51 &quot;kartikeyg0104&quot; was rejected (pending since 2026-02-19T18:48:39.707Z).
* 16:27 [[gitlab:renovatebot|@renovatebot]] was approved.
* 16:06 [[gitlab:gkm563|@gkm563]] was approved.

=== 2026-05-20 ===
* 01:21 &quot;beedellrokejulianlockhart&quot; was rejected (pending since 2026-02-18T01:19:13.284Z).

=== 2026-05-18 ===
* 23:18 &quot;wladek92&quot; was rejected (pending since 2026-02-16T23:16:22.939Z).
* 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved.

=== 2026-05-14 ===
* 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved.

=== 2026-05-13 ===
* 10:51 &quot;ssssaaaa&quot; was rejected (pending since 2026-02-11T10:50:36.975Z).

=== 2026-05-12 ===
* 18:06 [[gitlab:psubhashish|@psubhashish]] was approved.
* 08:12 &quot;khan&quot; was rejected (pending since 2026-02-10T08:11:48.776Z).
* 04:27 &quot;galaxysh&quot; was rejected (pending since 2026-02-10T04:24:59.440Z).

=== 2026-05-11 ===
* 12:18 &quot;peterxy12&quot; was rejected (pending since 2026-02-09T12:18:01.982Z).

=== 2026-05-10 ===
* 11:09 &quot;yalihupokn&quot; was rejected (pending since 2026-02-08T11:06:51.336Z).
* 05:12 &quot;wobadha&quot; was rejected (pending since 2026-02-08T05:11:00.569Z).

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>ertv87exmjsoqti0wlnhpml7l78cstq</sha1>
    </revision>
    <revision>
      <id>2435525</id>
      <parentid>2435494</parentid>
      <timestamp>2026-07-12T09:27:23Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>@smk was approved.</comment>
      <origin>2435525</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="60746" sha1="8pog1p7n1s4rza57yz2z979ezjdbal0" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-07-12 ===
* 09:27 [[gitlab:smk|@smk]] was approved.

=== 2026-07-11 ===
* 14:48 &quot;bigcereal42&quot; was rejected (pending since 2026-04-11T14:47:40.321Z).
* 12:18 &quot;pratyushsawan&quot; was rejected (pending since 2026-04-11T12:16:41.671Z).

=== 2026-07-10 ===
* 14:42 [[gitlab:akaza24|@akaza24]] was approved.
* 12:57 [[gitlab:kormisk|@kormisk]] was approved.

=== 2026-07-07 ===
* 10:36 [[gitlab:olafjanssen|@olafjanssen]] was approved.
* 06:57 &quot;elisapoly-99&quot; was rejected (pending since 2026-04-07T06:55:52.662Z).

=== 2026-07-06 ===
* 11:57 &quot;ma3rouf&quot; was rejected (pending since 2026-04-06T11:56:30.978Z).

=== 2026-07-02 ===
* 15:48 [[gitlab:tekneos|@tekneos]] was approved.

=== 2026-07-01 ===
* 15:12 [[gitlab:mugurolevy|@mugurolevy]] was approved.
* 14:15 [[gitlab:vadymts1|@vadymts1]] was approved.
* 09:57 &quot;mugurolevy&quot; was rejected (pending since 2026-04-01T09:55:19.175Z).

=== 2026-06-30 ===
* 14:27 &quot;shivangisharma&quot; was rejected (pending since 2026-03-31T14:26:44.932Z).

=== 2026-06-29 ===
* 19:03 [[gitlab:thisismattmiller|@thisismattmiller]] was approved.

=== 2026-06-28 ===
* 14:51 &quot;nkwenuinadine&quot; was rejected (pending since 2026-03-29T14:48:32.735Z).
* 14:03 &quot;vaishnavikumbhar&quot; was rejected (pending since 2026-03-29T14:01:30.604Z).
* 13:03 &quot;stepmay&quot; was rejected (pending since 2026-03-29T13:01:59.905Z).
* 06:51 &quot;swallroth&quot; was rejected (pending since 2026-03-29T06:49:54.838Z).

=== 2026-06-26 ===
* 09:39 [[gitlab:lakshita28|@lakshita28]] was approved.
* 07:30 [[gitlab:reeti|@reeti]] was approved.
* 07:30 [[gitlab:anushka10patel|@anushka10patel]] was approved.
* 07:30 &quot;samsaesque&quot; was rejected (pending since 2026-03-27T07:29:57.279Z).
* 05:51 [[gitlab:arpithhhaaa|@arpithhhaaa]] was approved.
* 05:51 [[gitlab:govindlaltl|@govindlaltl]] was approved.

=== 2026-06-25 ===
* 16:09 [[gitlab:sakuraemad|@sakuraemad]] was approved.
* 07:00 &quot;kdh8219&quot; was rejected (pending since 2026-03-26T06:58:05.415Z).

=== 2026-06-24 ===
* 11:54 [[gitlab:sanskardubeydev|@sanskardubeydev]] was approved.
* 10:09 &quot;tanmay789q&quot; was rejected (pending since 2026-03-25T10:07:54.602Z).

=== 2026-06-22 ===
* 19:57 [[gitlab:gouvernathor|@gouvernathor]] was approved.
* 16:45 [[gitlab:lucasbelo|@lucasbelo]] was approved.
* 07:15 &quot;jason2000-cpu&quot; was rejected (pending since 2026-03-23T07:14:09.184Z).

=== 2026-06-21 ===
* 13:18 [[gitlab:egonw|@egonw]] was approved.

=== 2026-06-20 ===
* 10:21 [[gitlab:tways2017|@tways2017]] was approved.

=== 2026-06-19 ===
* 16:06 &quot;wilsonwang2026&quot; was rejected (pending since 2026-03-20T16:06:05.511Z).
* 04:12 [[gitlab:claudio|@claudio]] was approved.

=== 2026-06-18 ===
* 14:21 &quot;royiswariii&quot; was rejected (pending since 2026-03-19T14:19:16.896Z).
* 13:06 [[gitlab:laurabarluzzi|@laurabarluzzi]] was approved.

=== 2026-06-17 ===
* 11:24 &quot;adinathq8x&quot; was rejected (pending since 2026-03-18T11:22:50.098Z).
* 09:45 &quot;nathanveritas&quot; was rejected (pending since 2026-03-18T09:43:51.645Z).

=== 2026-06-15 ===
* 22:39 [[gitlab:mohammadhijjawi|@mohammadhijjawi]] was approved.
* 14:24 &quot;enlisar&quot; was rejected (pending since 2026-03-16T14:23:00.109Z).
* 14:06 &quot;ayaan&quot; was rejected (pending since 2026-03-16T14:03:31.071Z).
* 10:54 &quot;kwametech&quot; was rejected (pending since 2026-03-16T10:54:11.083Z).

=== 2026-06-14 ===
* 17:45 [[gitlab:surajseth520|@surajseth520]] was approved.
* 07:24 &quot;malahimhaseeb&quot; was rejected (pending since 2026-03-15T07:21:57.748Z).

=== 2026-06-11 ===
* 11:48 [[gitlab:cadddr|@cadddr]] was approved.
* 11:18 &quot;wikipiggy&quot; was rejected (pending since 2026-03-12T11:16:09.335Z).
* 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved.

=== 2026-06-10 ===
* 07:03 [[gitlab:dmiranda|@dmiranda]] was approved.

=== 2026-06-09 ===
* 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved.
* 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved.
* 12:51 [[gitlab:ekrem|@ekrem]] was approved.

=== 2026-06-08 ===
* 17:48 &quot;jmprax&quot; was rejected (pending since 2026-03-09T17:46:38.807Z).
* 16:15 [[gitlab:ahonc|@ahonc]] was approved.
* 12:57 [[gitlab:rainmonger|@rainmonger]] was approved.

=== 2026-06-07 ===
* 23:03 &quot;shadowthewuff&quot; was rejected (pending since 2026-03-08T23:00:53.442Z).
* 11:45 &quot;wiki-pavan&quot; was rejected (pending since 2026-03-08T11:45:11.116Z).
* 02:39 [[gitlab:launchpad|@launchpad]] was approved.

=== 2026-06-06 ===
* 14:54 &quot;unicord&quot; was rejected (pending since 2026-03-07T14:52:04.992Z).
* 12:48 &quot;chien&quot; was rejected (pending since 2026-03-07T12:48:11.669Z).

=== 2026-06-04 ===
* 14:33 &quot;only-vikas&quot; was rejected (pending since 2026-03-05T14:32:09.186Z).

=== 2026-06-03 ===
* 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved.

=== 2026-06-02 ===
* 21:21 &quot;mgagat&quot; was rejected (pending since 2026-03-03T21:18:37.223Z).
* 13:57 &quot;prasunaenumarthy&quot; was rejected (pending since 2026-03-03T13:57:14.847Z).
* 05:48 [[gitlab:tmoney|@tmoney]] was approved.

=== 2026-06-01 ===
* 14:57 &quot;vikram2101&quot; was rejected (pending since 2026-03-02T14:54:26.550Z).
* 12:03 &quot;watshell&quot; was rejected (pending since 2026-03-02T12:03:09.329Z).

=== 2026-05-29 ===
* 12:48 &quot;mounikapotladurthi&quot; was rejected (pending since 2026-02-27T12:45:38.609Z).

=== 2026-05-27 ===
* 20:00 &quot;vinitha&quot; was rejected (pending since 2026-02-25T19:58:43.524Z).
* 16:30 &quot;codeurluce&quot; was rejected (pending since 2026-02-25T16:28:53.973Z).
* 14:33 [[gitlab:thilio|@thilio]] was approved.

=== 2026-05-26 ===
* 12:09 &quot;charisad&quot; was rejected (pending since 2026-02-24T12:07:21.881Z).

=== 2026-05-25 ===
* 22:54 &quot;ddshelto&quot; was rejected (pending since 2026-02-23T22:52:44.427Z).
* 19:51 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).
* 19:48 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).

=== 2026-05-24 ===
* 18:45 &quot;jiyagupta-cs&quot; was rejected (pending since 2026-02-22T18:43:33.176Z).

=== 2026-05-23 ===
* 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved.
* 04:21 [[gitlab:staraction|@staraction]] was approved.

=== 2026-05-22 ===
* 19:03 &quot;i-horich&quot; was rejected (pending since 2026-02-20T19:00:43.519Z).
* 01:48 &quot;50323233&quot; was rejected (pending since 2026-02-20T01:48:05.555Z).

=== 2026-05-21 ===
* 18:51 &quot;kartikeyg0104&quot; was rejected (pending since 2026-02-19T18:48:39.707Z).
* 16:27 [[gitlab:renovatebot|@renovatebot]] was approved.
* 16:06 [[gitlab:gkm563|@gkm563]] was approved.

=== 2026-05-20 ===
* 01:21 &quot;beedellrokejulianlockhart&quot; was rejected (pending since 2026-02-18T01:19:13.284Z).

=== 2026-05-18 ===
* 23:18 &quot;wladek92&quot; was rejected (pending since 2026-02-16T23:16:22.939Z).
* 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved.

=== 2026-05-14 ===
* 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved.

=== 2026-05-13 ===
* 10:51 &quot;ssssaaaa&quot; was rejected (pending since 2026-02-11T10:50:36.975Z).

=== 2026-05-12 ===
* 18:06 [[gitlab:psubhashish|@psubhashish]] was approved.
* 08:12 &quot;khan&quot; was rejected (pending since 2026-02-10T08:11:48.776Z).
* 04:27 &quot;galaxysh&quot; was rejected (pending since 2026-02-10T04:24:59.440Z).

=== 2026-05-11 ===
* 12:18 &quot;peterxy12&quot; was rejected (pending since 2026-02-09T12:18:01.982Z).

=== 2026-05-10 ===
* 11:09 &quot;yalihupokn&quot; was rejected (pending since 2026-02-08T11:06:51.336Z).
* 05:12 &quot;wobadha&quot; was rejected (pending since 2026-02-08T05:11:00.569Z).

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>8pog1p7n1s4rza57yz2z979ezjdbal0</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:Saimawnkham</title>
    <ns>3</ns>
    <id>456955</id>
    <revision>
      <id>2435498</id>
      <parentid>2269332</parentid>
      <timestamp>2026-07-11T18:08:14Z</timestamp>
      <contributor>
        <username>StrikerBot</username>
        <id>8475</id>
      </contributor>
      <comment>Welcome to Toolforge!</comment>
      <origin>2435498</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1138" sha1="78t1i5c0vnb50jutusudggng7c4fz6j" xml:space="preserve">== Wikitech account renamed and attached to SUL ==

Your Wikitech account has been renamed to match the SUL account you associated
it with using toolsadmin.wikimedia.org or idm.wikimedia.org. Following the
rename your Wikitech account was attached to your SUL account. You should now
be able to login to wikitech.wikimedia.org using your SUL account in the same
way you would login to any other Wikimedia project wiki. -- [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]]) 17:37, 10 February 2025 (UTC)
== Welcome to Toolforge! ==
Hello Saimawnkham, welcome to the Toolforge project!  Your request for access was processed, and you should be able to use ssh to connect to &lt;tt&gt;login.toolforge.org&lt;/tt&gt; and create and manage tools at https://toolsadmin.wikimedia.org/.

Check the [[Help:Toolforge|Toolforge help page]] for tips on using your account.  You can also ask questions in our IRC channel at {{irc|wikimedia-cloud}} or send an e-mail to our mailing list &lt;tt&gt;cloud@lists.wikimedia.org&lt;/tt&gt;.

Thank you, and have fun making Tools! --[[User:StrikerBot|StrikerBot]] ([[User talk:StrikerBot|talk]]) 18:08, 11 July 2026 (UTC)</text>
      <sha1>78t1i5c0vnb50jutusudggng7c4fz6j</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebot-syncer/SAL</title>
    <ns>498</ns>
    <id>459263</id>
    <revision>
      <id>2435496</id>
      <parentid>2433022</parentid>
      <timestamp>2026-07-11T17:46:41Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)</comment>
      <origin>2435496</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="5327" sha1="o4iccghrj3d1g9vc6tc6nn4gfa2bb3p" xml:space="preserve">=== 2026-07-11 ===
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>o4iccghrj3d1g9vc6tc6nn4gfa2bb3p</sha1>
    </revision>
    <revision>
      <id>2435497</id>
      <parentid>2435496</parentid>
      <timestamp>2026-07-11T18:06:47Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)</comment>
      <origin>2435497</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="5533" sha1="7q9blkkvfci1wu1ofavnyde09449pg7" xml:space="preserve">=== 2026-07-11 ===
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>7q9blkkvfci1wu1ofavnyde09449pg7</sha1>
    </revision>
    <revision>
      <id>2435499</id>
      <parentid>2435497</parentid>
      <timestamp>2026-07-11T18:09:17Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)</comment>
      <origin>2435499</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="5739" sha1="9139bf6i8x088v86bchqpgh9k0zomff" xml:space="preserve">=== 2026-07-11 ===
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>9139bf6i8x088v86bchqpgh9k0zomff</sha1>
    </revision>
    <revision>
      <id>2435501</id>
      <parentid>2435499</parentid>
      <timestamp>2026-07-11T19:43:08Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)</comment>
      <origin>2435501</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="5945" sha1="8idmojpnrdfaq4m5nzzfqradmudc42l" xml:space="preserve">=== 2026-07-11 ===
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>8idmojpnrdfaq4m5nzzfqradmudc42l</sha1>
    </revision>
    <revision>
      <id>2435502</id>
      <parentid>2435501</parentid>
      <timestamp>2026-07-11T19:49:11Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)</comment>
      <origin>2435502</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6154" sha1="0xoi4l6t9y8015y1q0gzl1h620cirqt" xml:space="preserve">=== 2026-07-11 ===
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>0xoi4l6t9y8015y1q0gzl1h620cirqt</sha1>
    </revision>
    <revision>
      <id>2435503</id>
      <parentid>2435502</parentid>
      <timestamp>2026-07-11T19:59:19Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)</comment>
      <origin>2435503</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6360" sha1="e009tkiumjjyz005tdwvgl58sp074oh" xml:space="preserve">=== 2026-07-11 ===
* 19:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>e009tkiumjjyz005tdwvgl58sp074oh</sha1>
    </revision>
    <revision>
      <id>2435504</id>
      <parentid>2435503</parentid>
      <timestamp>2026-07-11T20:02:42Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)</comment>
      <origin>2435504</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6566" sha1="ajc1z56a123ho8cprmuuxq2clvr3m8x" xml:space="preserve">=== 2026-07-11 ===
* 20:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ajc1z56a123ho8cprmuuxq2clvr3m8x</sha1>
    </revision>
    <revision>
      <id>2435505</id>
      <parentid>2435504</parentid>
      <timestamp>2026-07-11T20:04:50Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)</comment>
      <origin>2435505</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6775" sha1="ilcyt60lk6jyihoya4qm311hhepbofw" xml:space="preserve">=== 2026-07-11 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 20:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ilcyt60lk6jyihoya4qm311hhepbofw</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:Etherpad-backup/Log</title>
    <ns>116</ns>
    <id>460070</id>
    <revision>
      <id>2435493</id>
      <parentid>2433434</parentid>
      <timestamp>2026-07-11T14:02:09Z</timestamp>
      <contributor>
        <username>EtherpadBackupBot</username>
        <id>54504</id>
      </contributor>
      <comment>WikimediaEvents archived by Exec8</comment>
      <origin>2435493</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="193238" sha1="e9n1vteqode24yc46ntog878u1dwo6j" xml:space="preserve">=== 2026-07-11 ===
* 14:02 [[etherpad:p/WikimediaEvents]] was archived as [[etherpadbackup:WikimediaEvents]] by [[User:Exec8|Exec8]]

=== 2026-07-06 ===
* 21:30 [[etherpad:p/MWStake]] was archived as [[etherpadbackup:MWStake]] by [[User:Pppery|Pppery]]

=== 2026-06-21 ===
* 15:11 [[etherpad:p/IN-KL_Meetup_April_2026]] was archived as [[etherpadbackup:IN-KL_Meetup_April_2026]] by [[User:Gnoeee|Gnoeee]]
* 15:11 [[etherpad:p/IN-KL_Meetup_May_2026]] was archived as [[etherpadbackup:IN-KL_Meetup_May_2026]] by [[User:Gnoeee|Gnoeee]]

=== 2026-06-15 ===
* 09:55 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Ericliu1912|Ericliu1912]]

=== 2026-06-13 ===
* 12:08 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Chlod|Chlod]]

=== 2026-06-08 ===
* 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]]
* 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]]

=== 2026-05-29 ===
* 03:47 [[etherpad:p/ESEAPCon26_sessions_csv]] was archived as [[etherpadbackup:ESEAPCon26_sessions_csv]] by [[User:Robertsky|Robertsky]]
* 03:41 [[etherpad:p/ESEAPCon26_day2_Room2_DB9UHD]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_DB9UHD]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_JD3H99]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_JD3H99]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_DBKL7W]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_DBKL7W]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_3KZYPS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_3KZYPS]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_YF7VEZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_YF7VEZ]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_8TXHAF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_8TXHAF]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_QRT8BV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_QRT8BV]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_RFAU8M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_RFAU8M]] by [[User:Robertsky|Robertsky]]
* 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_VU3QL8]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_VU3QL8]] by [[User:Robertsky|Robertsky]]
* 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_M3QUZD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_M3QUZD]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_TFFBBH]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_TFFBBH]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_PQYYHB]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_PQYYHB]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_CA7G3X]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_CA7G3X]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_8RX8NZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_8RX8NZ]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_JYNFKE]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_JYNFKE]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_LFV39K]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_LFV39K]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_H9JCWN]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_H9JCWN]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_9EF8DZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9EF8DZ]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_8EQWQL]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_8EQWQL]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_A87WBV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_A87WBV]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_3K3T9M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_3K3T9M]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_9DJJP7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9DJJP7]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_AKYUKC]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_AKYUKC]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_S99FWD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_S99FWD]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_HGHY9E]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_HGHY9E]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_YL8ST7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_YL8ST7]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_8ZNVWS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_8ZNVWS]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_C7ZRET]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_C7ZRET]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_NCHDPG]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_NCHDPG]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_TW9WFJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_TW9WFJ]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_VNX3UJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_VNX3UJ]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_BQHSPF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_BQHSPF]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_ZNUEAR]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_ZNUEAR]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day2_Outofvenue_VVR7KR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_VVR7KR]] by [[User:Robertsky|Robertsky]]
* 03:29 [[etherpad:p/ESEAPCon26_day2_Outofvenue_CJTRQM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_CJTRQM]] by [[User:Robertsky|Robertsky]]
* 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_DEXKVE]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DEXKVE]] by [[User:Robertsky|Robertsky]]
* 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_7DSNAX]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7DSNAX]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_DZGGRB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DZGGRB]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_7MBXLG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7MBXLG]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_Q3XQYZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_Q3XQYZ]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_TDXWE3]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_TDXWE3]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_HZZZKL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_HZZZKL]] by [[User:Robertsky|Robertsky]]
* 03:27 [[etherpad:p/ESEAPCon26_day2_Room2_JTBXUG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JTBXUG]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_ALULGS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_ALULGS]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_MNBFWL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_MNBFWL]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_RRHPKR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_RRHPKR]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_AURUVZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_AURUVZ]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_3MAKHH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_3MAKHH]] by [[User:Robertsky|Robertsky]]
* 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_TXFRZB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_TXFRZB]] by [[User:Robertsky|Robertsky]]
* 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_QUFR97]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QUFR97]] by [[User:Robertsky|Robertsky]]
* 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_JT7TFH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JT7TFH]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_LJ3BL8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_LJ3BL8]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_9YDQCT]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_9YDQCT]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_GKN9F7]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_GKN9F7]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_CFUMZ9]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CFUMZ9]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_CRTN93]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CRTN93]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_MGZMP8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_MGZMP8]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_8WQJKB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_8WQJKB]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LLPZSB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LLPZSB]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_HMSJPM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_HMSJPM]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_SWCDHM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_SWCDHM]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_3CDL7X]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_3CDL7X]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_AHCNHB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_AHCNHB]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LTSVEH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LTSVEH]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_ZVYUWS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_ZVYUWS]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_T8YSUB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_T8YSUB]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_LMXU7L]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LMXU7L]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_YKMM9C]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_YKMM9C]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_RZKSWJ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_RZKSWJ]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Outofvenue_3RMDEL]] was archived as [[etherpadbackup:ESEAPCon26_day1_Outofvenue_3RMDEL]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room3_XAJUWY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room3_XAJUWY]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_FSZJK7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FSZJK7]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_W3ND3N]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_W3ND3N]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_8AR9U9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_8AR9U9]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_NUWQDM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_NUWQDM]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_DGZT88]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_DGZT88]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_FPWESV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FPWESV]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room1_7EKDB9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_7EKDB9]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_XVFLQU]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_XVFLQU]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_KCR737]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_KCR737]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_EBATCZ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_EBATCZ]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_MGKRNF]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_MGKRNF]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_DHFDW7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_DHFDW7]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Mainroom_CLMDUY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_CLMDUY]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_B7Y3PM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_B7Y3PM]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_WGT9AQ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_WGT9AQ]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_AXG3KB]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_AXG3KB]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_L8GRH9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_L8GRH9]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_UAF8RA]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_UAF8RA]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_NTXNZV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_NTXNZV]] by [[User:Robertsky|Robertsky]]

=== 2026-05-26 ===
* 21:47 [[etherpad:p/Wikimedia_Hackathon_2026_Closing_Showcase]] was archived as [[etherpadbackup:Wikimedia_Hackathon_2026_Closing_Showcase]] by [[User:Gnoeee|Gnoeee]]
* 21:47 [[etherpad:p/South_Asia_Community_Call]] was archived as [[etherpadbackup:South_Asia_Community_Call]] by [[User:Gnoeee|Gnoeee]]

=== 2026-05-23 ===
* 09:18 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Ericliu1912|Ericliu1912]]
* 09:18 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Ericliu1912|Ericliu1912]]
* 02:46 [[etherpad:p/org_structure_reading_group]] was archived as [[etherpadbackup:org_structure_reading_group]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-18 ===
* 19:20 [[etherpad:p/Chartres]] was archived as [[etherpadbackup:Chartres]] by [[User:Sj|Sj]]
* 19:08 [[etherpad:p/city-homelessness-okn]] was archived as [[etherpadbackup:city-homelessness-okn]] by [[User:Sj|Sj]]
* 19:07 [[etherpad:p/WM-structural-problems-overview]] was archived as [[etherpadbackup:WM-structural-problems-overview]] by [[User:Sj|Sj]]
* 19:03 [[etherpad:p/WLM-2021-May]] was archived as [[etherpadbackup:WLM-2021-May]] by [[User:Sj|Sj]]
* 19:02 [[etherpad:p/WLM-2021-April]] was archived as [[etherpadbackup:WLM-2021-April]] by [[User:Sj|Sj]]
* 19:02 [[etherpad:p/shexcg_minutes]] was archived as [[etherpadbackup:shexcg_minutes]] by [[User:Sj|Sj]]
* 19:01 [[etherpad:p/ctp-letter]] was archived as [[etherpadbackup:ctp-letter]] by [[User:Sj|Sj]]
* 18:59 [[etherpad:p/Wikimania_oral_history]] was archived as [[etherpadbackup:Wikimania_oral_history]] by [[User:Sj|Sj]]
* 18:55 [[etherpad:p/covid19]] was archived as [[etherpadbackup:covid19]] by [[User:Sj|Sj]]
* 18:54 [[etherpad:p/Covering_George_Floyd]] was archived as [[etherpadbackup:Covering_George_Floyd]] by [[User:Sj|Sj]]
* 17:37 [[etherpad:p/JROSTS18-HACKDAY]] was archived as [[etherpadbackup:JROSTS18-HACKDAY]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFF]] was archived as [[etherpadbackup:JROSTS18-UNCFF]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFE]] was archived as [[etherpadbackup:JROSTS18-UNCFE]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFD]] was archived as [[etherpadbackup:JROSTS18-UNCFD]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFC]] was archived as [[etherpadbackup:JROSTS18-UNCFC]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFB]] was archived as [[etherpadbackup:JROSTS18-UNCFB]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-OA.2]] was archived as [[etherpadbackup:JROSTS18-OA.2]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCONFERENCING]] was archived as [[etherpadbackup:JROSTS18-UNCONFERENCING]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-JROST]] was archived as [[etherpadbackup:JROSTS18-JROST]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-OB.1]] was archived as [[etherpadbackup:JROSTS18-OB.1]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-EOA]] was archived as [[etherpadbackup:JROSTS18-EOA]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-WFC.3]] was archived as [[etherpadbackup:JROSTS18-WFC.3]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-WFC.2]] was archived as [[etherpadbackup:JROSTS18-WFC.2]] by [[User:Sj|Sj]]
* 17:34 [[etherpad:p/JROSTS18-WFA.2]] was archived as [[etherpadbackup:JROSTS18-WFA.2]] by [[User:Sj|Sj]]
* 17:34 [[etherpad:p/JROSTS18-WFA.1]] was archived as [[etherpadbackup:JROSTS18-WFA.1]] by [[User:Sj|Sj]]
* 17:33 [[etherpad:p/JROSTS18-CROSSCUTTING]] was archived as [[etherpadbackup:JROSTS18-CROSSCUTTING]] by [[User:Sj|Sj]]
* 17:33 [[etherpad:p/JROST-Master]] was archived as [[etherpadbackup:JROST-Master]] by [[User:Sj|Sj]]
* 17:33 [[etherpad:p/JROSTS18-WFC.1]] was archived as [[etherpadbackup:JROSTS18-WFC.1]] by [[User:Sj|Sj]]
* 17:19 [[etherpad:p/15.320]] was archived as [[etherpadbackup:15.320]] by [[User:Sj|Sj]]
* 17:17 [[etherpad:p/medical_hotspots]] was archived as [[etherpadbackup:medical_hotspots]] by [[User:Sj|Sj]]
* 17:16 [[etherpad:p/community_health]] was archived as [[etherpadbackup:community_health]] by [[User:Sj|Sj]]
* 17:14 [[etherpad:p/whatsnext]] was archived as [[etherpadbackup:whatsnext]] by [[User:Sj|Sj]]
* 17:12 [[etherpad:p/MWF20160318]] was archived as [[etherpadbackup:MWF20160318]] by [[User:Sj|Sj]]
* 17:12 [[etherpad:p/MWF20160226]] was archived as [[etherpadbackup:MWF20160226]] by [[User:Sj|Sj]]
* 17:11 [[etherpad:p/ceem14-pr]] was archived as [[etherpadbackup:ceem14-pr]] by [[User:Sj|Sj]]
* 16:54 [[etherpad:p/Templates,_Page_Components_and_editing]] was archived as [[etherpadbackup:Templates,_Page_Components_and_editing]] by [[User:Sj|Sj]]
* 16:54 [[etherpad:p/GLAM-Wikipedia-fr]] was archived as [[etherpadbackup:GLAM-Wikipedia-fr]] by [[User:Sj|Sj]]
* 16:49 [[etherpad:p/editcontentmodel]] was archived as [[etherpadbackup:editcontentmodel]] by [[User:Sj|Sj]]
* 16:43 [[etherpad:p/multimedia-weekly-meeting-2014-05-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-05-14]] by [[User:Sj|Sj]]
* 16:06 [[etherpad:p/cuny-wikiwed-2026-04]] was archived as [[etherpadbackup:cuny-wikiwed-2026-04]] by [[User:Pharos|Pharos]]
* 16:06 [[etherpad:p/cuny-wikiwed-2026-03]] was archived as [[etherpadbackup:cuny-wikiwed-2026-03]] by [[User:Pharos|Pharos]]
* 16:06 [[etherpad:p/cuny-wikiwed-2026-01]] was archived as [[etherpadbackup:cuny-wikiwed-2026-01]] by [[User:Pharos|Pharos]]
* 16:06 [[etherpad:p/cuny-wikiwed-2025-11]] was archived as [[etherpadbackup:cuny-wikiwed-2025-11]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-10]] was archived as [[etherpadbackup:cuny-wikiwed-2025-10]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-09]] was archived as [[etherpadbackup:cuny-wikiwed-2025-09]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-08]] was archived as [[etherpadbackup:cuny-wikiwed-2025-08]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-07]] was archived as [[etherpadbackup:cuny-wikiwed-2025-07]] by [[User:Pharos|Pharos]]
* 16:04 [[etherpad:p/cuny-wikiwed-2025-06]] was archived as [[etherpadbackup:cuny-wikiwed-2025-06]] by [[User:Pharos|Pharos]]
* 16:04 [[etherpad:p/cuny-wikiwed-2025-05]] was archived as [[etherpadbackup:cuny-wikiwed-2025-05]] by [[User:Pharos|Pharos]]
* 16:04 [[etherpad:p/cuny-wikiwed-2025-04]] was archived as [[etherpadbackup:cuny-wikiwed-2025-04]] by [[User:Pharos|Pharos]]

=== 2026-05-10 ===
* 18:10 [[etherpad:p/wmhackshowcase2020]] was archived as [[etherpadbackup:wmhackshowcase2020]] by [[User:BryanDavis|BryanDavis]]
* 18:03 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 18:01 [[etherpad:p/wdqs-500s]] was archived as [[etherpadbackup:wdqs-500s]] by [[User:BryanDavis|BryanDavis]]
* 18:00 [[etherpad:p/wep_data_task_force_meeting_#1]] was archived as [[etherpadbackup:wep_data_task_force_meeting_#1]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-07 ===
* 00:35 [[etherpad:p/zh-ve-screenshots]] was archived as [[etherpadbackup:zh-ve-screenshots]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/zerocrash]] was archived as [[etherpadbackup:zerocrash]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/zero-mf-sync-up]] was archived as [[etherpadbackup:zero-mf-sync-up]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/yuvidump]] was archived as [[etherpadbackup:yuvidump]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/yuvi-knowledge-transfer]] was archived as [[etherpadbackup:yuvi-knowledge-transfer]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/yeZfmYkMu2]] was archived as [[etherpadbackup:yeZfmYkMu2]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/yAqOuwkDKuhwhwjvd44j]] was archived as [[etherpadbackup:yAqOuwkDKuhwhwjvd44j]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/y4gqz6&lt;fe46Q]] was archived as [[etherpadbackup:y4gqz6&lt;fe46Q]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/y4gqz6&lt;fe46Q]] was archived as [[etherpadbackup:y4gqz6&lt;fe46Q]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/xZLrPEASWV]] was archived as [[etherpadbackup:xZLrPEASWV]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/x-analytics]] was archived as [[etherpadbackup:x-analytics]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/wtf-is-production]] was archived as [[etherpadbackup:wtf-is-production]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/wtf]] was archived as [[etherpadbackup:wtf]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/wp-app-1-1-release-post]] was archived as [[etherpadbackup:wp-app-1-1-release-post]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/wmhack2021-phab]] was archived as [[etherpadbackup:wmhack2021-phab]] by [[User:BryanDavis|BryanDavis]]
* 00:27 [[etherpad:p/wmhack2017-ciw]] was archived as [[etherpadbackup:wmhack2017-ciw]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/wmhack15-wikidata-architecture-session]] was archived as [[etherpadbackup:wmhack15-wikidata-architecture-session]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/wmh2024-leaving-coordination]] was archived as [[etherpadbackup:wmh2024-leaving-coordination]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/wmh2023-Toolhub,_Toolhunt,_and_the_quest]] was archived as [[etherpadbackup:wmh2023-Toolhub,_Toolhunt,_and_the_quest]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/wmh2023-Overview_of_technical_areas_%26_projects]] was archived as [[etherpadbackup:wmh2023-Overview_of_technical_areas_%26_projects]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] was archived as [[etherpadbackup:wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] by [[User:BryanDavis|BryanDavis]]
* 00:23 [[etherpad:p/wmh2023-Best_practices_&amp;_ideas_for_implementing]] was archived as [[etherpadbackup:wmh2023-Best_practices_&amp;_ideas_for_implementing]] by [[User:BryanDavis|BryanDavis]]
* 00:23 [[etherpad:p/wmf11and12]] was archived as [[etherpadbackup:wmf11and12]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/wmcslightning]] was archived as [[etherpadbackup:wmcslightning]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/wmcs_nfs_future]] was archived as [[etherpadbackup:wmcs_nfs_future]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/wmcs-vs-rowb-upgrade]] was archived as [[etherpadbackup:wmcs-vs-rowb-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/wmcs-goals-q4-16-17]] was archived as [[etherpadbackup:wmcs-goals-q4-16-17]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/wmcs-goals-q2-17-18]] was archived as [[etherpadbackup:wmcs-goals-q2-17-18]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/wmcs-fallout]] was archived as [[etherpadbackup:wmcs-fallout]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/wmcs-cumin-fail]] was archived as [[etherpadbackup:wmcs-cumin-fail]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/wmcs-contributors]] was archived as [[etherpadbackup:wmcs-contributors]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/wmcs-2023-06-07]] was archived as [[etherpadbackup:wmcs-2023-06-07]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/wmconf2013-evaluating_programs]] was archived as [[etherpadbackup:wmconf2013-evaluating_programs]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/wmconf2013-evaluating-programs]] was archived as [[etherpadbackup:wmconf2013-evaluating-programs]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/wlmapp-iteration1-story-estimation-and-review]] was archived as [[etherpadbackup:wlmapp-iteration1-story-estimation-and-review]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/wlm-sprint5-retro]] was archived as [[etherpadbackup:wlm-sprint5-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/wlm-sprint4-retro]] was archived as [[etherpadbackup:wlm-sprint4-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/wlm-sprint3-retro]] was archived as [[etherpadbackup:wlm-sprint3-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/wlm-sprint2-retro]] was archived as [[etherpadbackup:wlm-sprint2-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:14 [[etherpad:p/wlm-sprint1-retro]] was archived as [[etherpadbackup:wlm-sprint1-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/wlm-project-retro]] was archived as [[etherpadbackup:wlm-project-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/wlm-on-labs-steps]] was archived as [[etherpadbackup:wlm-on-labs-steps]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/wlm-api-admin-tree]] was archived as [[etherpadbackup:wlm-api-admin-tree]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/wishathon-march-2024-wishlist-overview-session]] was archived as [[etherpadbackup:wishathon-march-2024-wishlist-overview-session]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/wishathon-march-2024-showcase]] was archived as [[etherpadbackup:wishathon-march-2024-showcase]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/wishathon-march-2024-office-hour]] was archived as [[etherpadbackup:wishathon-march-2024-office-hour]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/wishathon-march-2024-fotw-session]] was archived as [[etherpadbackup:wishathon-march-2024-fotw-session]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/wikitechrebuild]] was archived as [[etherpadbackup:wikitechrebuild]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/wikitech-extensions]] was archived as [[etherpadbackup:wikitech-extensions]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/wikireplica-dns-switch]] was archived as [[etherpadbackup:wikireplica-dns-switch]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/wikireplica-beta]] was archived as [[etherpadbackup:wikireplica-beta]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/wikipedia-mobile-1-2-beta1-notes]] was archived as [[etherpadbackup:wikipedia-mobile-1-2-beta1-notes]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/wikipedia-ios-4.0.5-release-notes]] was archived as [[etherpadbackup:wikipedia-ios-4.0.5-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/wikipedia-app-saved-pages-architecture]] was archived as [[etherpadbackup:wikipedia-app-saved-pages-architecture]] by [[User:BryanDavis|BryanDavis]]
* 00:06 [[etherpad:p/wikipedia-app-ios-issues]] was archived as [[etherpadbackup:wikipedia-app-ios-issues]] by [[User:BryanDavis|BryanDavis]]
* 00:06 [[etherpad:p/wikimania_knowledge]] was archived as [[etherpadbackup:wikimania_knowledge]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/wikigroknextsteps]] was archived as [[etherpadbackup:wikigroknextsteps]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/wikigrokcollections]] was archived as [[etherpadbackup:wikigrokcollections]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/wikidev17-devwishlist]] was archived as [[etherpadbackup:wikidev17-devwishlist]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/wikidata_query_service_checkin]] was archived as [[etherpadbackup:wikidata_query_service_checkin]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/wikibits-undefault]] was archived as [[etherpadbackup:wikibits-undefault]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/wiki_technology_notes]] was archived as [[etherpadbackup:wiki_technology_notes]] by [[User:BryanDavis|BryanDavis]]
* 00:02 [[etherpad:p/widgets-for-uploadwizard]] was archived as [[etherpadbackup:widgets-for-uploadwizard]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/whm2023-Overview_of_technical_areas_&amp;_projects]] was archived as [[etherpadbackup:whm2023-Overview_of_technical_areas_&amp;_projects]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/whats-left-in-toolforge-eqiad]] was archived as [[etherpadbackup:whats-left-in-toolforge-eqiad]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/what_is_left_in_tampa]] was archived as [[etherpadbackup:what_is_left_in_tampa]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/webservice-k8s-announce]] was archived as [[etherpadbackup:webservice-k8s-announce]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-06 ===
* 23:59 [[etherpad:p/webrequest-2021-Q4-loss]] was archived as [[etherpadbackup:webrequest-2021-Q4-loss]] by [[User:BryanDavis|BryanDavis]]
* 23:59 [[etherpad:p/webreq_update]] was archived as [[etherpadbackup:webreq_update]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/we-really-love-nfs]] was archived as [[etherpadbackup:we-really-love-nfs]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/wdqs_graph_split_production_deploy]] was archived as [[etherpadbackup:wdqs_graph_split_production_deploy]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/wdqs-split-graph-rollout-plan]] was archived as [[etherpadbackup:wdqs-split-graph-rollout-plan]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/wdqs-2024-08-08]] was archived as [[etherpadbackup:wdqs-2024-08-08]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/wdio]] was archived as [[etherpadbackup:wdio]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/wcna_tools]] was archived as [[etherpadbackup:wcna_tools]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/wbq_test]] was archived as [[etherpadbackup:wbq_test]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/watchlist_id]] was archived as [[etherpadbackup:watchlist_id]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/wallaby-dev-magnum-install]] was archived as [[etherpadbackup:wallaby-dev-magnum-install]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/volans-tmp]] was archived as [[etherpadbackup:volans-tmp]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/vk-jumbo-cleanup]] was archived as [[etherpadbackup:vk-jumbo-cleanup]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/virt1006migrate]] was archived as [[etherpadbackup:virt1006migrate]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/vikassy]] was archived as [[etherpadbackup:vikassy]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/vikas_ve]] was archived as [[etherpadbackup:vikas_ve]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/vikas_vagrant]] was archived as [[etherpadbackup:vikas_vagrant]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/vikas-upload-error]] was archived as [[etherpadbackup:vikas-upload-error]] by [[User:BryanDavis|BryanDavis]]
* 23:50 [[etherpad:p/vikas-ruby-style]] was archived as [[etherpadbackup:vikas-ruby-style]] by [[User:BryanDavis|BryanDavis]]
* 23:49 [[etherpad:p/vikas-gerrit]] was archived as [[etherpadbackup:vikas-gerrit]] by [[User:BryanDavis|BryanDavis]]
* 23:49 [[etherpad:p/vikas]] was archived as [[etherpadbackup:vikas]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/ve-future-thoughts-2015]] was archived as [[etherpadbackup:ve-future-thoughts-2015]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/ve-2016-05-05]] was archived as [[etherpadbackup:ve-2016-05-05]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/ve]] was archived as [[etherpadbackup:ve]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/varnishkafka_troubleshooting]] was archived as [[etherpadbackup:varnishkafka_troubleshooting]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/vandrew]] was archived as [[etherpadbackup:vandrew]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/uw-cucumber-sketch]] was archived as [[etherpadbackup:uw-cucumber-sketch]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/uw-bugday-2014-09-09]] was archived as [[etherpadbackup:uw-bugday-2014-09-09]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/uploadwizard-refactor-planning]] was archived as [[etherpadbackup:uploadwizard-refactor-planning]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/uploadwizard-browser-tests]] was archived as [[etherpadbackup:uploadwizard-browser-tests]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/uploadcampaigns-squids]] was archived as [[etherpadbackup:uploadcampaigns-squids]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/upload-tool-class-structure]] was archived as [[etherpadbackup:upload-tool-class-structure]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/unsophabricated]] was archived as [[etherpadbackup:unsophabricated]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/unittestingbrownbag]] was archived as [[etherpadbackup:unittestingbrownbag]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/unified]] was archived as [[etherpadbackup:unified]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/unaliased]] was archived as [[etherpadbackup:unaliased]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/unable_to_move_translated_page]] was archived as [[etherpadbackup:unable_to_move_translated_page]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/twocolconflictdeploy]] was archived as [[etherpadbackup:twocolconflictdeploy]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/twiterbot-debuging]] was archived as [[etherpadbackup:twiterbot-debuging]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/trusty]] was archived as [[etherpadbackup:trusty]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/trixie-announcement]] was archived as [[etherpadbackup:trixie-announcement]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/trend-prod]] was archived as [[etherpadbackup:trend-prod]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/trebuchet-propose]] was archived as [[etherpadbackup:trebuchet-propose]] by [[User:BryanDavis|BryanDavis]]
* 23:37 [[etherpad:p/toomanythings]] was archived as [[etherpadbackup:toomanythings]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/toolssupport]] was archived as [[etherpadbackup:toolssupport]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/toolsdb-upgrade]] was archived as [[etherpadbackup:toolsdb-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/toolsdb-10.6]] was archived as [[etherpadbackup:toolsdb-10.6]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/tools-trusty-move]] was archived as [[etherpadbackup:tools-trusty-move]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/tools-the-great-recompress]] was archived as [[etherpadbackup:tools-the-great-recompress]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/tools-reboots-cve-0728]] was archived as [[etherpadbackup:tools-reboots-cve-0728]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/tools-reboots]] was archived as [[etherpadbackup:tools-reboots]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/tools-puppet-prefixes]] was archived as [[etherpadbackup:tools-puppet-prefixes]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/tools-migration-nodes]] was archived as [[etherpadbackup:tools-migration-nodes]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/tools-migration]] was archived as [[etherpadbackup:tools-migration]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/tools-kubernetes-scenarios]] was archived as [[etherpadbackup:tools-kubernetes-scenarios]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/tools-k8s-chat-01]] was archived as [[etherpadbackup:tools-k8s-chat-01]] by [[User:BryanDavis|BryanDavis]]
* 23:30 [[etherpad:p/toollabs20160127]] was archived as [[etherpadbackup:toollabs20160127]] by [[User:BryanDavis|BryanDavis]]
* 23:30 [[etherpad:p/toollabs-webservice-trusty-switch]] was archived as [[etherpadbackup:toollabs-webservice-trusty-switch]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/toollabs-uwsgi]] was archived as [[etherpadbackup:toollabs-uwsgi]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/toollabs-quarter-goal-q4-2014-15]] was archived as [[etherpadbackup:toollabs-quarter-goal-q4-2014-15]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/toollabs-impact]] was archived as [[etherpadbackup:toollabs-impact]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/toollabs-ideal-getting-started]] was archived as [[etherpadbackup:toollabs-ideal-getting-started]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/toollabs-cdnjs]] was archived as [[etherpadbackup:toollabs-cdnjs]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/toolforge-reboots]] was archived as [[etherpadbackup:toolforge-reboots]] by [[User:BryanDavis|BryanDavis]]
* 23:26 [[etherpad:p/toolforge-php72-webservice]] was archived as [[etherpadbackup:toolforge-php72-webservice]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/toolforge-k8s-upgrade-1.23]] was archived as [[etherpadbackup:toolforge-k8s-upgrade-1.23]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/toolforge-k8s-1.31]] was archived as [[etherpadbackup:toolforge-k8s-1.31]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/toolforge-k8s-1.28-to-1.29-upgrade]] was archived as [[etherpadbackup:toolforge-k8s-1.28-to-1.29-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/toolforge-2021-12-14]] was archived as [[etherpadbackup:toolforge-2021-12-14]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/tool_labs-doc]] was archived as [[etherpadbackup:tool_labs-doc]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/tonythomas01]] was archived as [[etherpadbackup:tonythomas01]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/tofu]] was archived as [[etherpadbackup:tofu]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/todays-regressions]] was archived as [[etherpadbackup:todays-regressions]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/title_value]] was archived as [[etherpadbackup:title_value]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/things_to_restart_after_rabbit_explosion]] was archived as [[etherpadbackup:things_to_restart_after_rabbit_explosion]] by [[User:BryanDavis|BryanDavis]]
* 06:03 [[etherpad:p/test]] was archived as [[etherpadbackup:test]] by [[User:BryanDavis|BryanDavis]]
* 06:03 [[etherpad:p/termdata]] was archived as [[etherpadbackup:termdata]] by [[User:BryanDavis|BryanDavis]]
* 06:02 [[etherpad:p/temp]] was archived as [[etherpadbackup:temp]] by [[User:BryanDavis|BryanDavis]]
* 06:02 [[etherpad:p/teach-yuvi-to-walk]] was archived as [[etherpadbackup:teach-yuvi-to-walk]] by [[User:BryanDavis|BryanDavis]]
* 06:01 [[etherpad:p/tcinterviewquestions]] was archived as [[etherpadbackup:tcinterviewquestions]] by [[User:BryanDavis|BryanDavis]]
* 06:00 [[etherpad:p/tallinntransportation]] was archived as [[etherpadbackup:tallinntransportation]] by [[User:BryanDavis|BryanDavis]]
* 06:00 [[etherpad:p/tabsinpuppet]] was archived as [[etherpadbackup:tabsinpuppet]] by [[User:BryanDavis|BryanDavis]]
* 05:59 [[etherpad:p/tFyLAurf0Z]] was archived as [[etherpadbackup:tFyLAurf0Z]] by [[User:BryanDavis|BryanDavis]]
* 05:59 [[etherpad:p/switchover]] was archived as [[etherpadbackup:switchover]] by [[User:BryanDavis|BryanDavis]]
* 05:58 [[etherpad:p/swiftfail]] was archived as [[etherpadbackup:swiftfail]] by [[User:BryanDavis|BryanDavis]]
* 05:58 [[etherpad:p/swhYPuS04T]] was archived as [[etherpadbackup:swhYPuS04T]] by [[User:BryanDavis|BryanDavis]]
* 05:57 [[etherpad:p/supersetdeprecation]] was archived as [[etherpadbackup:supersetdeprecation]] by [[User:BryanDavis|BryanDavis]]
* 05:57 [[etherpad:p/sulf]] was archived as [[etherpadbackup:sulf]] by [[User:BryanDavis|BryanDavis]]
* 05:56 [[etherpad:p/styleguide]] was archived as [[etherpadbackup:styleguide]] by [[User:BryanDavis|BryanDavis]]
* 05:56 [[etherpad:p/stuffs]] was archived as [[etherpadbackup:stuffs]] by [[User:BryanDavis|BryanDavis]]
* 05:55 [[etherpad:p/stubsExpansion]] was archived as [[etherpadbackup:stubsExpansion]] by [[User:BryanDavis|BryanDavis]]
* 05:55 [[etherpad:p/striker-prod-deploy]] was archived as [[etherpadbackup:striker-prod-deploy]] by [[User:BryanDavis|BryanDavis]]
* 05:54 [[etherpad:p/striker-gitlab-20220906]] was archived as [[etherpadbackup:striker-gitlab-20220906]] by [[User:BryanDavis|BryanDavis]]
* 05:54 [[etherpad:p/striker]] was archived as [[etherpadbackup:striker]] by [[User:BryanDavis|BryanDavis]]
* 05:53 [[etherpad:p/streaming_updater_cutover]] was archived as [[etherpadbackup:streaming_updater_cutover]] by [[User:BryanDavis|BryanDavis]]
* 05:53 [[etherpad:p/streaming-wdqs]] was archived as [[etherpadbackup:streaming-wdqs]] by [[User:BryanDavis|BryanDavis]]
* 05:52 [[etherpad:p/stillnopms]] was archived as [[etherpadbackup:stillnopms]] by [[User:BryanDavis|BryanDavis]]
* 05:52 [[etherpad:p/state_of_union_-_if_-network_bullet_points]] was archived as [[etherpadbackup:state_of_union_-_if_-network_bullet_points]] by [[User:BryanDavis|BryanDavis]]
* 05:51 [[etherpad:p/stat1_accounts]] was archived as [[etherpadbackup:stat1_accounts]] by [[User:BryanDavis|BryanDavis]]
* 05:51 [[etherpad:p/stat-analytics-vlan]] was archived as [[etherpadbackup:stat-analytics-vlan]] by [[User:BryanDavis|BryanDavis]]
* 05:50 [[etherpad:p/standalone_masters]] was archived as [[etherpadbackup:standalone_masters]] by [[User:BryanDavis|BryanDavis]]
* 05:50 [[etherpad:p/ssl]] was archived as [[etherpadbackup:ssl]] by [[User:BryanDavis|BryanDavis]]
* 05:49 [[etherpad:p/ssh-key-change-trixie]] was archived as [[etherpadbackup:ssh-key-change-trixie]] by [[User:BryanDavis|BryanDavis]]
* 05:48 [[etherpad:p/spark_pair_coding]] was archived as [[etherpadbackup:spark_pair_coding]] by [[User:BryanDavis|BryanDavis]]
* 05:48 [[etherpad:p/spark2-planning]] was archived as [[etherpadbackup:spark2-planning]] by [[User:BryanDavis|BryanDavis]]
* 05:47 [[etherpad:p/so1920.cs.unibo.it]] was archived as [[etherpadbackup:so1920.cs.unibo.it]] by [[User:BryanDavis|BryanDavis]]
* 05:47 [[etherpad:p/sistersearch-AB-test]] was archived as [[etherpadbackup:sistersearch-AB-test]] by [[User:BryanDavis|BryanDavis]]
* 05:46 [[etherpad:p/single-edit-tab]] was archived as [[etherpadbackup:single-edit-tab]] by [[User:BryanDavis|BryanDavis]]
* 05:46 [[etherpad:p/shit_tim_says]] was archived as [[etherpadbackup:shit_tim_says]] by [[User:BryanDavis|BryanDavis]]
* 05:45 [[etherpad:p/setting_up_ORES_lab_cluster]] was archived as [[etherpadbackup:setting_up_ORES_lab_cluster]] by [[User:BryanDavis|BryanDavis]]
* 05:45 [[etherpad:p/sendBeacon-2014-10-03]] was archived as [[etherpadbackup:sendBeacon-2014-10-03]] by [[User:BryanDavis|BryanDavis]]
* 05:44 [[etherpad:p/selenium]] was archived as [[etherpadbackup:selenium]] by [[User:BryanDavis|BryanDavis]]
* 05:44 [[etherpad:p/security_group_tests]] was archived as [[etherpadbackup:security_group_tests]] by [[User:BryanDavis|BryanDavis]]
* 05:43 [[etherpad:p/search_enhancements]] was archived as [[etherpadbackup:search_enhancements]] by [[User:BryanDavis|BryanDavis]]
* 05:43 [[etherpad:p/search-logging]] was archived as [[etherpadbackup:search-logging]] by [[User:BryanDavis|BryanDavis]]
* 05:42 [[etherpad:p/search-hypotesis-24-25]] was archived as [[etherpadbackup:search-hypotesis-24-25]] by [[User:BryanDavis|BryanDavis]]
* 05:42 [[etherpad:p/search-503s]] was archived as [[etherpadbackup:search-503s]] by [[User:BryanDavis|BryanDavis]]
* 05:41 [[etherpad:p/scrum-of-scrums]] was archived as [[etherpadbackup:scrum-of-scrums]] by [[User:BryanDavis|BryanDavis]]
* 05:41 [[etherpad:p/scoping-troubles]] was archived as [[etherpadbackup:scoping-troubles]] by [[User:BryanDavis|BryanDavis]]
* 05:40 [[etherpad:p/scap-commands]] was archived as [[etherpadbackup:scap-commands]] by [[User:BryanDavis|BryanDavis]]
* 05:40 [[etherpad:p/scap-201617-q2]] was archived as [[etherpadbackup:scap-201617-q2]] by [[User:BryanDavis|BryanDavis]]
* 05:39 [[etherpad:p/rubocop-2015-02-10]] was archived as [[etherpadbackup:rubocop-2015-02-10]] by [[User:BryanDavis|BryanDavis]]
* 05:39 [[etherpad:p/rsyncnova]] was archived as [[etherpadbackup:rsyncnova]] by [[User:BryanDavis|BryanDavis]]
* 05:38 [[etherpad:p/rollout-ldap-block-change]] was archived as [[etherpadbackup:rollout-ldap-block-change]] by [[User:BryanDavis|BryanDavis]]
* 05:38 [[etherpad:p/role-mediainfo]] was archived as [[etherpadbackup:role-mediainfo]] by [[User:BryanDavis|BryanDavis]]
* 05:37 [[etherpad:p/rockyupgrade]] was archived as [[etherpadbackup:rockyupgrade]] by [[User:BryanDavis|BryanDavis]]
* 05:37 [[etherpad:p/robots.txt]] was archived as [[etherpadbackup:robots.txt]] by [[User:BryanDavis|BryanDavis]]
* 05:36 [[etherpad:p/rmrebootsforcephmons]] was archived as [[etherpadbackup:rmrebootsforcephmons]] by [[User:BryanDavis|BryanDavis]]
* 05:35 [[etherpad:p/revscoring_user_oriented]] was archived as [[etherpadbackup:revscoring_user_oriented]] by [[User:BryanDavis|BryanDavis]]
* 05:35 [[etherpad:p/revscoring_process]] was archived as [[etherpadbackup:revscoring_process]] by [[User:BryanDavis|BryanDavis]]
* 05:34 [[etherpad:p/revscoring_languages]] was archived as [[etherpadbackup:revscoring_languages]] by [[User:BryanDavis|BryanDavis]]
* 05:34 [[etherpad:p/revscoring_hackathon]] was archived as [[etherpadbackup:revscoring_hackathon]] by [[User:BryanDavis|BryanDavis]]
* 05:33 [[etherpad:p/review-workflow-yuvi]] was archived as [[etherpadbackup:review-workflow-yuvi]] by [[User:BryanDavis|BryanDavis]]
* 05:33 [[etherpad:p/resurrect_omega]] was archived as [[etherpadbackup:resurrect_omega]] by [[User:BryanDavis|BryanDavis]]
* 05:32 [[etherpad:p/resthack]] was archived as [[etherpadbackup:resthack]] by [[User:BryanDavis|BryanDavis]]
* 05:32 [[etherpad:p/restbase-parsoid-storage-rollout]] was archived as [[etherpadbackup:restbase-parsoid-storage-rollout]] by [[User:BryanDavis|BryanDavis]]
* 05:31 [[etherpad:p/researchers_in_our_midst]] was archived as [[etherpadbackup:researchers_in_our_midst]] by [[User:BryanDavis|BryanDavis]]
* 05:31 [[etherpad:p/research_showcase_dec14]] was archived as [[etherpadbackup:research_showcase_dec14]] by [[User:BryanDavis|BryanDavis]]
* 05:30 [[etherpad:p/replicas-temp-accounts]] was archived as [[etherpadbackup:replicas-temp-accounts]] by [[User:BryanDavis|BryanDavis]]
* 05:30 [[etherpad:p/rep-lag-s8]] was archived as [[etherpadbackup:rep-lag-s8]] by [[User:BryanDavis|BryanDavis]]
* 05:29 [[etherpad:p/remaining-ldap]] was archived as [[etherpadbackup:remaining-ldap]] by [[User:BryanDavis|BryanDavis]]
* 05:29 [[etherpad:p/refinery_deploy]] was archived as [[etherpadbackup:refinery_deploy]] by [[User:BryanDavis|BryanDavis]]
* 05:28 [[etherpad:p/red-links]] was archived as [[etherpadbackup:red-links]] by [[User:BryanDavis|BryanDavis]]
* 05:28 [[etherpad:p/realm_networks]] was archived as [[etherpadbackup:realm_networks]] by [[User:BryanDavis|BryanDavis]]
* 05:27 [[etherpad:p/reading-infra-2015-16-q4]] was archived as [[etherpadbackup:reading-infra-2015-16-q4]] by [[User:BryanDavis|BryanDavis]]
* 05:27 [[etherpad:p/rdf-flink-k8s]] was archived as [[etherpadbackup:rdf-flink-k8s]] by [[User:BryanDavis|BryanDavis]]
* 05:26 [[etherpad:p/rdbmess-2021]] was archived as [[etherpadbackup:rdbmess-2021]] by [[User:BryanDavis|BryanDavis]]
* 05:26 [[etherpad:p/rcstream-misc]] was archived as [[etherpadbackup:rcstream-misc]] by [[User:BryanDavis|BryanDavis]]
* 05:25 [[etherpad:p/rc1-release-email]] was archived as [[etherpadbackup:rc1-release-email]] by [[User:BryanDavis|BryanDavis]]
* 05:25 [[etherpad:p/rc.2-announce]] was archived as [[etherpadbackup:rc.2-announce]] by [[User:BryanDavis|BryanDavis]]
* 05:24 [[etherpad:p/rar]] was archived as [[etherpadbackup:rar]] by [[User:BryanDavis|BryanDavis]]
* 05:24 [[etherpad:p/rada12dubna]] was archived as [[etherpadbackup:rada12dubna]] by [[User:BryanDavis|BryanDavis]]
* 05:23 [[etherpad:p/r.d2e7710af5a20eefd93babda3f38929a]] was archived as [[etherpadbackup:r.d2e7710af5a20eefd93babda3f38929a]] by [[User:BryanDavis|BryanDavis]]
* 05:22 [[etherpad:p/r.B0XlRhOdKRWT6xuH]] was archived as [[etherpadbackup:r.B0XlRhOdKRWT6xuH]] by [[User:BryanDavis|BryanDavis]]
* 05:22 [[etherpad:p/quicksurveys]] was archived as [[etherpadbackup:quicksurveys]] by [[User:BryanDavis|BryanDavis]]
* 05:21 [[etherpad:p/quibble-0.0.36]] was archived as [[etherpadbackup:quibble-0.0.36]] by [[User:BryanDavis|BryanDavis]]
* 05:21 [[etherpad:p/quarry-wdqs-integration]] was archived as [[etherpadbackup:quarry-wdqs-integration]] by [[User:BryanDavis|BryanDavis]]
* 05:20 [[etherpad:p/quarry-for-pwb-scripts]] was archived as [[etherpadbackup:quarry-for-pwb-scripts]] by [[User:BryanDavis|BryanDavis]]
* 05:20 [[etherpad:p/quarry-announce-email]] was archived as [[etherpadbackup:quarry-announce-email]] by [[User:BryanDavis|BryanDavis]]
* 05:19 [[etherpad:p/q_planning_retro]] was archived as [[etherpadbackup:q_planning_retro]] by [[User:BryanDavis|BryanDavis]]
* 05:19 [[etherpad:p/q4planningretro]] was archived as [[etherpadbackup:q4planningretro]] by [[User:BryanDavis|BryanDavis]]
* 05:18 [[etherpad:p/q4planningnotes]] was archived as [[etherpadbackup:q4planningnotes]] by [[User:BryanDavis|BryanDavis]]
* 05:18 [[etherpad:p/q4_checkin]] was archived as [[etherpadbackup:q4_checkin]] by [[User:BryanDavis|BryanDavis]]
* 05:17 [[etherpad:p/q2_eng_goal]] was archived as [[etherpadbackup:q2_eng_goal]] by [[User:BryanDavis|BryanDavis]]
* 05:17 [[etherpad:p/python_mediawiki_devs]] was archived as [[etherpadbackup:python_mediawiki_devs]] by [[User:BryanDavis|BryanDavis]]
* 05:16 [[etherpad:p/python]] was archived as [[etherpadbackup:python]] by [[User:BryanDavis|BryanDavis]]
* 05:16 [[etherpad:p/push-to-deploy-beta-announce]] was archived as [[etherpadbackup:push-to-deploy-beta-announce]] by [[User:BryanDavis|BryanDavis]]
* 05:15 [[etherpad:p/puppetswat]] was archived as [[etherpadbackup:puppetswat]] by [[User:BryanDavis|BryanDavis]]
* 05:15 [[etherpad:p/puppetmaster-enc]] was archived as [[etherpadbackup:puppetmaster-enc]] by [[User:BryanDavis|BryanDavis]]
* 05:14 [[etherpad:p/puppet_upgrade_vms]] was archived as [[etherpadbackup:puppet_upgrade_vms]] by [[User:BryanDavis|BryanDavis]]
* 05:14 [[etherpad:p/puppet3]] was archived as [[etherpadbackup:puppet3]] by [[User:BryanDavis|BryanDavis]]
* 05:13 [[etherpad:p/publish-new-mobile-report]] was archived as [[etherpadbackup:publish-new-mobile-report]] by [[User:BryanDavis|BryanDavis]]
* 05:13 [[etherpad:p/proxy-buster]] was archived as [[etherpadbackup:proxy-buster]] by [[User:BryanDavis|BryanDavis]]
* 05:12 [[etherpad:p/preciseonlabs]] was archived as [[etherpadbackup:preciseonlabs]] by [[User:BryanDavis|BryanDavis]]
* 05:12 [[etherpad:p/precise-tools]] was archived as [[etherpadbackup:precise-tools]] by [[User:BryanDavis|BryanDavis]]
* 05:11 [[etherpad:p/poem]] was archived as [[etherpadbackup:poem]] by [[User:BryanDavis|BryanDavis]]
* 05:11 [[etherpad:p/play_store_what's_new]] was archived as [[etherpadbackup:play_store_what's_new]] by [[User:BryanDavis|BryanDavis]]
* 05:10 [[etherpad:p/pii_pageviews]] was archived as [[etherpadbackup:pii_pageviews]] by [[User:BryanDavis|BryanDavis]]
* 05:09 [[etherpad:p/phab_roles]] was archived as [[etherpadbackup:phab_roles]] by [[User:BryanDavis|BryanDavis]]
* 05:09 [[etherpad:p/phab-2016-02-03]] was archived as [[etherpadbackup:phab-2016-02-03]] by [[User:BryanDavis|BryanDavis]]
* 05:08 [[etherpad:p/phab]] was archived as [[etherpadbackup:phab]] by [[User:BryanDavis|BryanDavis]]
* 05:08 [[etherpad:p/performance-team-weekly-update]] was archived as [[etherpadbackup:performance-team-weekly-update]] by [[User:BryanDavis|BryanDavis]]
* 05:07 [[etherpad:p/performance-k8s]] was archived as [[etherpadbackup:performance-k8s]] by [[User:BryanDavis|BryanDavis]]
* 05:07 [[etherpad:p/perfnotice]] was archived as [[etherpadbackup:perfnotice]] by [[User:BryanDavis|BryanDavis]]
* 05:06 [[etherpad:p/perf-announcements]] was archived as [[etherpadbackup:perf-announcements]] by [[User:BryanDavis|BryanDavis]]
* 05:06 [[etherpad:p/perf-20160223]] was archived as [[etherpadbackup:perf-20160223]] by [[User:BryanDavis|BryanDavis]]
* 05:05 [[etherpad:p/paws-tools]] was archived as [[etherpadbackup:paws-tools]] by [[User:BryanDavis|BryanDavis]]
* 05:05 [[etherpad:p/paws-public-url-structure]] was archived as [[etherpadbackup:paws-public-url-structure]] by [[User:BryanDavis|BryanDavis]]
* 05:04 [[etherpad:p/paste]] was archived as [[etherpadbackup:paste]] by [[User:BryanDavis|BryanDavis]]
* 05:04 [[etherpad:p/parsoid_docs]] was archived as [[etherpadbackup:parsoid_docs]] by [[User:BryanDavis|BryanDavis]]
* 05:03 [[etherpad:p/parsoid-2019-01-16]] was archived as [[etherpadbackup:parsoid-2019-01-16]] by [[User:BryanDavis|BryanDavis]]
* 05:03 [[etherpad:p/parsing_team_meeting_archive]] was archived as [[etherpadbackup:parsing_team_meeting_archive]] by [[User:BryanDavis|BryanDavis]]
* 05:02 [[etherpad:p/parser-api-2018]] was archived as [[etherpadbackup:parser-api-2018]] by [[User:BryanDavis|BryanDavis]]
* 05:02 [[etherpad:p/pageview_api_nodes]] was archived as [[etherpadbackup:pageview_api_nodes]] by [[User:BryanDavis|BryanDavis]]
* 05:01 [[etherpad:p/pages-needing-investigation]] was archived as [[etherpadbackup:pages-needing-investigation]] by [[User:BryanDavis|BryanDavis]]
* 05:01 [[etherpad:p/page-styling-zurich]] was archived as [[etherpadbackup:page-styling-zurich]] by [[User:BryanDavis|BryanDavis]]
* 05:00 [[etherpad:p/otrs-migration]] was archived as [[etherpadbackup:otrs-migration]] by [[User:BryanDavis|BryanDavis]]
* 05:00 [[etherpad:p/ores_task_tracking]] was archived as [[etherpadbackup:ores_task_tracking]] by [[User:BryanDavis|BryanDavis]]
* 04:59 [[etherpad:p/ores]] was archived as [[etherpadbackup:ores]] by [[User:BryanDavis|BryanDavis]]
* 04:58 [[etherpad:p/ops-offsite-discussions]] was archived as [[etherpadbackup:ops-offsite-discussions]] by [[User:BryanDavis|BryanDavis]]
* 04:58 [[etherpad:p/ops-offsite-2016-lightning-talks]] was archived as [[etherpadbackup:ops-offsite-2016-lightning-talks]] by [[User:BryanDavis|BryanDavis]]
* 04:57 [[etherpad:p/ops-offsite-2016-labstore-future]] was archived as [[etherpadbackup:ops-offsite-2016-labstore-future]] by [[User:BryanDavis|BryanDavis]]
* 04:57 [[etherpad:p/opensearch-opportunities-for-improvement]] was archived as [[etherpadbackup:opensearch-opportunities-for-improvement]] by [[User:BryanDavis|BryanDavis]]
* 04:56 [[etherpad:p/opendj-migration]] was archived as [[etherpadbackup:opendj-migration]] by [[User:BryanDavis|BryanDavis]]
* 04:56 [[etherpad:p/open_infra_workshop]] was archived as [[etherpadbackup:open_infra_workshop]] by [[User:BryanDavis|BryanDavis]]
* 04:55 [[etherpad:p/opcache-💩-2020-10-01]] was archived as [[etherpadbackup:opcache-💩-2020-10-01]] by [[User:BryanDavis|BryanDavis]]
* 04:55 [[etherpad:p/oozie-restart]] was archived as [[etherpadbackup:oozie-restart]] by [[User:BryanDavis|BryanDavis]]
* 04:54 [[etherpad:p/oozie]] was archived as [[etherpadbackup:oozie]] by [[User:BryanDavis|BryanDavis]]
* 04:54 [[etherpad:p/omega-restarts]] was archived as [[etherpadbackup:omega-restarts]] by [[User:BryanDavis|BryanDavis]]
* 04:53 [[etherpad:p/oldbastion]] was archived as [[etherpadbackup:oldbastion]] by [[User:BryanDavis|BryanDavis]]
* 04:53 [[etherpad:p/okapi-dumps-2021-02-01]] was archived as [[etherpadbackup:okapi-dumps-2021-02-01]] by [[User:BryanDavis|BryanDavis]]
* 04:52 [[etherpad:p/ocata-upgrade]] was archived as [[etherpadbackup:ocata-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 04:52 [[etherpad:p/o1Ub87jizX]] was archived as [[etherpadbackup:o1Ub87jizX]] by [[User:BryanDavis|BryanDavis]]
* 04:51 [[etherpad:p/o]] was archived as [[etherpadbackup:o]] by [[User:BryanDavis|BryanDavis]]
* 04:51 [[etherpad:p/nuria]] was archived as [[etherpadbackup:nuria]] by [[User:BryanDavis|BryanDavis]]
* 04:50 [[etherpad:p/npp_discussion_wmhack17]] was archived as [[etherpadbackup:npp_discussion_wmhack17]] by [[User:BryanDavis|BryanDavis]]
* 04:50 [[etherpad:p/nodes_with_a_public_IP]] was archived as [[etherpadbackup:nodes_with_a_public_IP]] by [[User:BryanDavis|BryanDavis]]
* 04:49 [[etherpad:p/nodepool-migration]] was archived as [[etherpadbackup:nodepool-migration]] by [[User:BryanDavis|BryanDavis]]
* 04:49 [[etherpad:p/night-mode-colors]] was archived as [[etherpadbackup:night-mode-colors]] by [[User:BryanDavis|BryanDavis]]
* 04:48 [[etherpad:p/ng8EYRhQ7SIXy7cLBexZ]] was archived as [[etherpadbackup:ng8EYRhQ7SIXy7cLBexZ]] by [[User:BryanDavis|BryanDavis]]
* 04:48 [[etherpad:p/nfsprojectchange]] was archived as [[etherpadbackup:nfsprojectchange]] by [[User:BryanDavis|BryanDavis]]
* 04:47 [[etherpad:p/nfsmaint]] was archived as [[etherpadbackup:nfsmaint]] by [[User:BryanDavis|BryanDavis]]
* 04:47 [[etherpad:p/nfs]] was archived as [[etherpadbackup:nfs]] by [[User:BryanDavis|BryanDavis]]
* 04:46 [[etherpad:p/newton_upgrade_plan]] was archived as [[etherpadbackup:newton_upgrade_plan]] by [[User:BryanDavis|BryanDavis]]
* 04:45 [[etherpad:p/newsletters-contenthandler]] was archived as [[etherpadbackup:newsletters-contenthandler]] by [[User:BryanDavis|BryanDavis]]
* 04:45 [[etherpad:p/newpyter]] was archived as [[etherpadbackup:newpyter]] by [[User:BryanDavis|BryanDavis]]
* 04:44 [[etherpad:p/new-webnode-toollabs]] was archived as [[etherpadbackup:new-webnode-toollabs]] by [[User:BryanDavis|BryanDavis]]
* 04:44 [[etherpad:p/networkoutage]] was archived as [[etherpadbackup:networkoutage]] by [[User:BryanDavis|BryanDavis]]
* 04:43 [[etherpad:p/netbox4-upgrade]] was archived as [[etherpadbackup:netbox4-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 04:43 [[etherpad:p/netbox-fail]] was archived as [[etherpadbackup:netbox-fail]] by [[User:BryanDavis|BryanDavis]]
* 04:42 [[etherpad:p/naming-visualization]] was archived as [[etherpadbackup:naming-visualization]] by [[User:BryanDavis|BryanDavis]]
* 04:42 [[etherpad:p/nTbAb5yPvBFATUylV7Vj]] was archived as [[etherpadbackup:nTbAb5yPvBFATUylV7Vj]] by [[User:BryanDavis|BryanDavis]]
* 04:41 [[etherpad:p/mwtidy-checkerrors]] was archived as [[etherpadbackup:mwtidy-checkerrors]] by [[User:BryanDavis|BryanDavis]]
* 04:41 [[etherpad:p/mwds15-spec-oriented-architecture]] was archived as [[etherpadbackup:mwds15-spec-oriented-architecture]] by [[User:BryanDavis|BryanDavis]]
* 04:40 [[etherpad:p/mwds-profiling]] was archived as [[etherpadbackup:mwds-profiling]] by [[User:BryanDavis|BryanDavis]]
* 04:40 [[etherpad:p/mw1-25-jsdeprecate]] was archived as [[etherpadbackup:mw1-25-jsdeprecate]] by [[User:BryanDavis|BryanDavis]]
* 04:39 [[etherpad:p/mw-php70-tests]] was archived as [[etherpadbackup:mw-php70-tests]] by [[User:BryanDavis|BryanDavis]]
* 04:39 [[etherpad:p/mvrelease]] was archived as [[etherpadbackup:mvrelease]] by [[User:BryanDavis|BryanDavis]]
* 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-12-24]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-12-24]] by [[User:BryanDavis|BryanDavis]]
* 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-01-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-01-14]] by [[User:BryanDavis|BryanDavis]]
* 04:37 [[etherpad:p/multimedia-weekly-meeting-2014-01-06]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-01-06]] by [[User:BryanDavis|BryanDavis]]
* 04:37 [[etherpad:p/multimedia-mmv-technical-retrospective]] was archived as [[etherpadbackup:multimedia-mmv-technical-retrospective]] by [[User:BryanDavis|BryanDavis]]
* 04:36 [[etherpad:p/multimedia-design-05-15-2014]] was archived as [[etherpadbackup:multimedia-design-05-15-2014]] by [[User:BryanDavis|BryanDavis]]
* 04:36 [[etherpad:p/multimedia-analytics-queries]] was archived as [[etherpadbackup:multimedia-analytics-queries]] by [[User:BryanDavis|BryanDavis]]
* 04:35 [[etherpad:p/morethan10g]] was archived as [[etherpadbackup:morethan10g]] by [[User:BryanDavis|BryanDavis]]
* 04:35 [[etherpad:p/monitoring-lvs-l2]] was archived as [[etherpadbackup:monitoring-lvs-l2]] by [[User:BryanDavis|BryanDavis]]
* 04:34 [[etherpad:p/mod-conf]] was archived as [[etherpadbackup:mod-conf]] by [[User:BryanDavis|BryanDavis]]
* 04:34 [[etherpad:p/mobileweb-wsj]] was archived as [[etherpadbackup:mobileweb-wsj]] by [[User:BryanDavis|BryanDavis]]
* 04:33 [[etherpad:p/mobilesprintname]] was archived as [[etherpadbackup:mobilesprintname]] by [[User:BryanDavis|BryanDavis]]
* 04:33 [[etherpad:p/mobilenav-sprint2-retro]] was archived as [[etherpadbackup:mobilenav-sprint2-retro]] by [[User:BryanDavis|BryanDavis]]
* 04:32 [[etherpad:p/mobilenav-sprint1-retro]] was archived as [[etherpadbackup:mobilenav-sprint1-retro]] by [[User:BryanDavis|BryanDavis]]
* 04:32 [[etherpad:p/mobileappplanningQ32014]] was archived as [[etherpadbackup:mobileappplanningQ32014]] by [[User:BryanDavis|BryanDavis]]
* 04:31 [[etherpad:p/mobile_web_q3]] was archived as [[etherpadbackup:mobile_web_q3]] by [[User:BryanDavis|BryanDavis]]
* 04:30 [[etherpad:p/mobile_web_health_check]] was archived as [[etherpadbackup:mobile_web_health_check]] by [[User:BryanDavis|BryanDavis]]
* 04:30 [[etherpad:p/mobile_web_design_review]] was archived as [[etherpadbackup:mobile_web_design_review]] by [[User:BryanDavis|BryanDavis]]
* 04:29 [[etherpad:p/mobile_web_breakout]] was archived as [[etherpadbackup:mobile_web_breakout]] by [[User:BryanDavis|BryanDavis]]
* 04:29 [[etherpad:p/mobile_apps_onboarding_brian]] was archived as [[etherpadbackup:mobile_apps_onboarding_brian]] by [[User:BryanDavis|BryanDavis]]
* 04:28 [[etherpad:p/mobile-users-faq]] was archived as [[etherpadbackup:mobile-users-faq]] by [[User:BryanDavis|BryanDavis]]
* 04:28 [[etherpad:p/mobile-tutorial-practice]] was archived as [[etherpadbackup:mobile-tutorial-practice]] by [[User:BryanDavis|BryanDavis]]
* 04:27 [[etherpad:p/mobile-retrospective-q3-2013]] was archived as [[etherpadbackup:mobile-retrospective-q3-2013]] by [[User:BryanDavis|BryanDavis]]
* 04:27 [[etherpad:p/mobile-retrospective-20130128]] was archived as [[etherpadbackup:mobile-retrospective-20130128]] by [[User:BryanDavis|BryanDavis]]
* 04:26 [[etherpad:p/mobile-retrospective-20130114]] was archived as [[etherpadbackup:mobile-retrospective-20130114]] by [[User:BryanDavis|BryanDavis]]
* 04:26 [[etherpad:p/mobile-retrospective-20121217]] was archived as [[etherpadbackup:mobile-retrospective-20121217]] by [[User:BryanDavis|BryanDavis]]
* 04:25 [[etherpad:p/mobile-retrospective-20120102]] was archived as [[etherpadbackup:mobile-retrospective-20120102]] by [[User:BryanDavis|BryanDavis]]
* 04:25 [[etherpad:p/mobile-q4-planning-2013]] was archived as [[etherpadbackup:mobile-q4-planning-2013]] by [[User:BryanDavis|BryanDavis]]
* 04:24 [[etherpad:p/mobile-q3-planning-2013]] was archived as [[etherpadbackup:mobile-q3-planning-2013]] by [[User:BryanDavis|BryanDavis]]
* 04:24 [[etherpad:p/mobile-ops-syncup-28feb2013]] was archived as [[etherpadbackup:mobile-ops-syncup-28feb2013]] by [[User:BryanDavis|BryanDavis]]
* 04:23 [[etherpad:p/mobile-ime-support]] was archived as [[etherpadbackup:mobile-ime-support]] by [[User:BryanDavis|BryanDavis]]
* 04:23 [[etherpad:p/mobile-campaigns]] was archived as [[etherpadbackup:mobile-campaigns]] by [[User:BryanDavis|BryanDavis]]
* 04:22 [[etherpad:p/mobile-app-workflow-el-1]] was archived as [[etherpadbackup:mobile-app-workflow-el-1]] by [[User:BryanDavis|BryanDavis]]
* 04:22 [[etherpad:p/mobile-app-versions]] was archived as [[etherpadbackup:mobile-app-versions]] by [[User:BryanDavis|BryanDavis]]
* 04:21 [[etherpad:p/mobile-app-otrs-feedback]] was archived as [[etherpadbackup:mobile-app-otrs-feedback]] by [[User:BryanDavis|BryanDavis]]
* 04:21 [[etherpad:p/mobile-app-dashboard]] was archived as [[etherpadbackup:mobile-app-dashboard]] by [[User:BryanDavis|BryanDavis]]
* 04:20 [[etherpad:p/mmv-text-size-changes]] was archived as [[etherpadbackup:mmv-text-size-changes]] by [[User:BryanDavis|BryanDavis]]
* 04:20 [[etherpad:p/mmodell]] was archived as [[etherpadbackup:mmodell]] by [[User:BryanDavis|BryanDavis]]
* 04:19 [[etherpad:p/mm-standup-notes]] was archived as [[etherpadbackup:mm-standup-notes]] by [[User:BryanDavis|BryanDavis]]
* 04:19 [[etherpad:p/misc-migration]] was archived as [[etherpadbackup:misc-migration]] by [[User:BryanDavis|BryanDavis]]
* 04:18 [[etherpad:p/migrate-k8s-etcd]] was archived as [[etherpadbackup:migrate-k8s-etcd]] by [[User:BryanDavis|BryanDavis]]
* 04:18 [[etherpad:p/mf]] was archived as [[etherpadbackup:mf]] by [[User:BryanDavis|BryanDavis]]
* 04:17 [[etherpad:p/mediawiki-vagrant]] was archived as [[etherpadbackup:mediawiki-vagrant]] by [[User:BryanDavis|BryanDavis]]
* 04:16 [[etherpad:p/mazza]] was archived as [[etherpadbackup:mazza]] by [[User:BryanDavis|BryanDavis]]
* 04:16 [[etherpad:p/mayankmadan]] was archived as [[etherpadbackup:mayankmadan]] by [[User:BryanDavis|BryanDavis]]
* 04:15 [[etherpad:p/mailman-Aug-2015]] was archived as [[etherpadbackup:mailman-Aug-2015]] by [[User:BryanDavis|BryanDavis]]
* 04:15 [[etherpad:p/magru_server_swaps]] was archived as [[etherpadbackup:magru_server_swaps]] by [[User:BryanDavis|BryanDavis]]
* 04:14 [[etherpad:p/magnumusersfeb2026]] was archived as [[etherpadbackup:magnumusersfeb2026]] by [[User:BryanDavis|BryanDavis]]
* 04:14 [[etherpad:p/mBBgYyJeBG]] was archived as [[etherpadbackup:mBBgYyJeBG]] by [[User:BryanDavis|BryanDavis]]
* 04:13 [[etherpad:p/m1-service-owners]] was archived as [[etherpadbackup:m1-service-owners]] by [[User:BryanDavis|BryanDavis]]
* 04:13 [[etherpad:p/lyon-service-template]] was archived as [[etherpadbackup:lyon-service-template]] by [[User:BryanDavis|BryanDavis]]
* 04:12 [[etherpad:p/lucid_mini_hack]] was archived as [[etherpadbackup:lucid_mini_hack]] by [[User:BryanDavis|BryanDavis]]
* 04:12 [[etherpad:p/lubaochuan]] was archived as [[etherpadbackup:lubaochuan]] by [[User:BryanDavis|BryanDavis]]
* 04:11 [[etherpad:p/logchange]] was archived as [[etherpadbackup:logchange]] by [[User:BryanDavis|BryanDavis]]
* 04:11 [[etherpad:p/linkpass]] was archived as [[etherpadbackup:linkpass]] by [[User:BryanDavis|BryanDavis]]
* 04:10 [[etherpad:p/legal-text-upload-dialog]] was archived as [[etherpadbackup:legal-text-upload-dialog]] by [[User:BryanDavis|BryanDavis]]
* 04:10 [[etherpad:p/language-and-localization]] was archived as [[etherpadbackup:language-and-localization]] by [[User:BryanDavis|BryanDavis]]
* 04:09 [[etherpad:p/langaugescreenshot-upload-error]] was archived as [[etherpadbackup:langaugescreenshot-upload-error]] by [[User:BryanDavis|BryanDavis]]
* 04:09 [[etherpad:p/lang-bar]] was archived as [[etherpadbackup:lang-bar]] by [[User:BryanDavis|BryanDavis]]
* 04:08 [[etherpad:p/labwebneedsports]] was archived as [[etherpadbackup:labwebneedsports]] by [[User:BryanDavis|BryanDavis]]
* 04:08 [[etherpad:p/labweb]] was archived as [[etherpadbackup:labweb]] by [[User:BryanDavis|BryanDavis]]
* 04:07 [[etherpad:p/labvirt_reboots]] was archived as [[etherpadbackup:labvirt_reboots]] by [[User:BryanDavis|BryanDavis]]
* 04:07 [[etherpad:p/labvirt1015]] was archived as [[etherpadbackup:labvirt1015]] by [[User:BryanDavis|BryanDavis]]
* 04:06 [[etherpad:p/labvirt1003instances]] was archived as [[etherpadbackup:labvirt1003instances]] by [[User:BryanDavis|BryanDavis]]
* 04:06 [[etherpad:p/labtes_mitaka_upgrade_notes]] was archived as [[etherpadbackup:labtes_mitaka_upgrade_notes]] by [[User:BryanDavis|BryanDavis]]
* 04:05 [[etherpad:p/labstore10045-move]] was archived as [[etherpadbackup:labstore10045-move]] by [[User:BryanDavis|BryanDavis]]
* 04:04 [[etherpad:p/labservices_failover]] was archived as [[etherpadbackup:labservices_failover]] by [[User:BryanDavis|BryanDavis]]
* 04:04 [[etherpad:p/labsdb10067]] was archived as [[etherpadbackup:labsdb10067]] by [[User:BryanDavis|BryanDavis]]
* 04:03 [[etherpad:p/labsdb-replication]] was archived as [[etherpadbackup:labsdb-replication]] by [[User:BryanDavis|BryanDavis]]
* 04:03 [[etherpad:p/labsdb-goal]] was archived as [[etherpadbackup:labsdb-goal]] by [[User:BryanDavis|BryanDavis]]
* 04:02 [[etherpad:p/labsdb-chat-july-18]] was archived as [[etherpadbackup:labsdb-chat-july-18]] by [[User:BryanDavis|BryanDavis]]
* 04:02 [[etherpad:p/labs_migration]] was archived as [[etherpadbackup:labs_migration]] by [[User:BryanDavis|BryanDavis]]
* 04:01 [[etherpad:p/labs-survey-interview-questions]] was archived as [[etherpadbackup:labs-survey-interview-questions]] by [[User:BryanDavis|BryanDavis]]
* 04:01 [[etherpad:p/labs-report-q4-1]] was archived as [[etherpadbackup:labs-report-q4-1]] by [[User:BryanDavis|BryanDavis]]
* 04:00 [[etherpad:p/labs-releng-k8s-confluence]] was archived as [[etherpadbackup:labs-releng-k8s-confluence]] by [[User:BryanDavis|BryanDavis]]
* 04:00 [[etherpad:p/labs-nfs-20140413-nfsoutage]] was archived as [[etherpadbackup:labs-nfs-20140413-nfsoutage]] by [[User:BryanDavis|BryanDavis]]
* 03:59 [[etherpad:p/labs-migration]] was archived as [[etherpadbackup:labs-migration]] by [[User:BryanDavis|BryanDavis]]
* 03:59 [[etherpad:p/labs-meeting-apr-25-2016]] was archived as [[etherpadbackup:labs-meeting-apr-25-2016]] by [[User:BryanDavis|BryanDavis]]
* 03:58 [[etherpad:p/labs-incident-timeline]] was archived as [[etherpadbackup:labs-incident-timeline]] by [[User:BryanDavis|BryanDavis]]
* 03:58 [[etherpad:p/labs-cleanup-2015]] was archived as [[etherpadbackup:labs-cleanup-2015]] by [[User:BryanDavis|BryanDavis]]
* 03:57 [[etherpad:p/labpuppetmaster1001-stragglers]] was archived as [[etherpadbackup:labpuppetmaster1001-stragglers]] by [[User:BryanDavis|BryanDavis]]
* 03:57 [[etherpad:p/labnetfailover]] was archived as [[etherpadbackup:labnetfailover]] by [[User:BryanDavis|BryanDavis]]
* 03:56 [[etherpad:p/l10nupdate]] was archived as [[etherpadbackup:l10nupdate]] by [[User:BryanDavis|BryanDavis]]
* 03:56 [[etherpad:p/l10n-translate-centralnotice]] was archived as [[etherpadbackup:l10n-translate-centralnotice]] by [[User:BryanDavis|BryanDavis]]
* 03:55 [[etherpad:p/l10n-team-2013-10]] was archived as [[etherpadbackup:l10n-team-2013-10]] by [[User:BryanDavis|BryanDavis]]
* 03:55 [[etherpad:p/kubecon-2017-offsite-agenda]] was archived as [[etherpadbackup:kubecon-2017-offsite-agenda]] by [[User:BryanDavis|BryanDavis]]
* 03:54 [[etherpad:p/kubecon]] was archived as [[etherpadbackup:kubecon]] by [[User:BryanDavis|BryanDavis]]
* 03:54 [[etherpad:p/kubeadm]] was archived as [[etherpadbackup:kubeadm]] by [[User:BryanDavis|BryanDavis]]
* 03:53 [[etherpad:p/kill-gridengine-slowly]] was archived as [[etherpadbackup:kill-gridengine-slowly]] by [[User:BryanDavis|BryanDavis]]
* 03:53 [[etherpad:p/keystoneldap]] was archived as [[etherpadbackup:keystoneldap]] by [[User:BryanDavis|BryanDavis]]
* 03:52 [[etherpad:p/keystonebootstrap]] was archived as [[etherpadbackup:keystonebootstrap]] by [[User:BryanDavis|BryanDavis]]
* 03:52 [[etherpad:p/kafkaswift]] was archived as [[etherpadbackup:kafkaswift]] by [[User:BryanDavis|BryanDavis]]
* 03:51 [[etherpad:p/kZaM0swM4c]] was archived as [[etherpadbackup:kZaM0swM4c]] by [[User:BryanDavis|BryanDavis]]
* 03:51 [[etherpad:p/kAh7UZSoyI]] was archived as [[etherpadbackup:kAh7UZSoyI]] by [[User:BryanDavis|BryanDavis]]
* 03:50 [[etherpad:p/kAKhhwCsJ7]] was archived as [[etherpadbackup:kAKhhwCsJ7]] by [[User:BryanDavis|BryanDavis]]
* 03:49 [[etherpad:p/k8s-status-wishlist]] was archived as [[etherpadbackup:k8s-status-wishlist]] by [[User:BryanDavis|BryanDavis]]
* 03:49 [[etherpad:p/jupyter-at-wikimedia]] was archived as [[etherpadbackup:jupyter-at-wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 03:48 [[etherpad:p/junk]] was archived as [[etherpadbackup:junk]] by [[User:BryanDavis|BryanDavis]]
* 03:48 [[etherpad:p/joes-random-notes]] was archived as [[etherpadbackup:joes-random-notes]] by [[User:BryanDavis|BryanDavis]]
* 03:47 [[etherpad:p/jmxtrans-packaging-improvements]] was archived as [[etherpadbackup:jmxtrans-packaging-improvements]] by [[User:BryanDavis|BryanDavis]]
* 03:47 [[etherpad:p/jjb]] was archived as [[etherpadbackup:jjb]] by [[User:BryanDavis|BryanDavis]]
* 03:46 [[etherpad:p/jenkins-upgrading]] was archived as [[etherpadbackup:jenkins-upgrading]] by [[User:BryanDavis|BryanDavis]]
* 03:46 [[etherpad:p/jenkins-job-builder]] was archived as [[etherpadbackup:jenkins-job-builder]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/javascript-scoping]] was archived as [[etherpadbackup:javascript-scoping]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/jagori]] was archived as [[etherpadbackup:jagori]] by [[User:BryanDavis|BryanDavis]]
* 03:44 [[etherpad:p/jQUprade]] was archived as [[etherpadbackup:jQUprade]] by [[User:BryanDavis|BryanDavis]]
* 03:44 [[etherpad:p/j9qaVR_wF9bIWn6BWuIr]] was archived as [[etherpadbackup:j9qaVR_wF9bIWn6BWuIr]] by [[User:BryanDavis|BryanDavis]]
* 03:43 [[etherpad:p/j4Obl63I8y]] was archived as [[etherpadbackup:j4Obl63I8y]] by [[User:BryanDavis|BryanDavis]]
* 03:43 [[etherpad:p/iterationname]] was archived as [[etherpadbackup:iterationname]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/is]] was archived as [[etherpadbackup:is]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/ipv6]] was archived as [[etherpadbackup:ipv6]] by [[User:BryanDavis|BryanDavis]]
* 03:41 [[etherpad:p/iosreleasing]] was archived as [[etherpadbackup:iosreleasing]] by [[User:BryanDavis|BryanDavis]]
* 03:41 [[etherpad:p/iosrelease]] was archived as [[etherpadbackup:iosrelease]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/ios_app_update_release_notes]] was archived as [[etherpadbackup:ios_app_update_release_notes]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/ios-5.0]] was archived as [[etherpadbackup:ios-5.0]] by [[User:BryanDavis|BryanDavis]]
* 03:39 [[etherpad:p/ios-3-1beta1-release-notes]] was archived as [[etherpadbackup:ios-3-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 03:39 [[etherpad:p/intro_to_revscores]] was archived as [[etherpadbackup:intro_to_revscores]] by [[User:BryanDavis|BryanDavis]]
* 03:38 [[etherpad:p/internal-graph-split-lvs]] was archived as [[etherpadbackup:internal-graph-split-lvs]] by [[User:BryanDavis|BryanDavis]]
* 03:38 [[etherpad:p/intercontinental-debugging-may-07-2012]] was archived as [[etherpadbackup:intercontinental-debugging-may-07-2012]] by [[User:BryanDavis|BryanDavis]]
* 03:37 [[etherpad:p/incident-20190509-codfwpuppetmasterdown]] was archived as [[etherpadbackup:incident-20190509-codfwpuppetmasterdown]] by [[User:BryanDavis|BryanDavis]]
* 03:37 [[etherpad:p/in_article_search_query]] was archived as [[etherpadbackup:in_article_search_query]] by [[User:BryanDavis|BryanDavis]]
* 03:36 [[etherpad:p/import_wcqs]] was archived as [[etherpadbackup:import_wcqs]] by [[User:BryanDavis|BryanDavis]]
* 03:36 [[etherpad:p/ihatesasl]] was archived as [[etherpadbackup:ihatesasl]] by [[User:BryanDavis|BryanDavis]]
* 03:35 [[etherpad:p/iOS_available_locale_ids]] was archived as [[etherpadbackup:iOS_available_locale_ids]] by [[User:BryanDavis|BryanDavis]]
* 03:34 [[etherpad:p/iOSDataMerge]] was archived as [[etherpadbackup:iOSDataMerge]] by [[User:BryanDavis|BryanDavis]]
* 03:34 [[etherpad:p/iOSAppi18nIssues]] was archived as [[etherpadbackup:iOSAppi18nIssues]] by [[User:BryanDavis|BryanDavis]]
* 03:33 [[etherpad:p/iOSAppDescription]] was archived as [[etherpadbackup:iOSAppDescription]] by [[User:BryanDavis|BryanDavis]]
* 03:33 [[etherpad:p/iOS-ChangeLog]] was archived as [[etherpadbackup:iOS-ChangeLog]] by [[User:BryanDavis|BryanDavis]]
* 03:32 [[etherpad:p/iOS-7-issues]] was archived as [[etherpadbackup:iOS-7-issues]] by [[User:BryanDavis|BryanDavis]]
* 03:32 [[etherpad:p/i8XjsnBxxh]] was archived as [[etherpadbackup:i8XjsnBxxh]] by [[User:BryanDavis|BryanDavis]]
* 03:31 [[etherpad:p/i18n-rfc-2013-11]] was archived as [[etherpadbackup:i18n-rfc-2013-11]] by [[User:BryanDavis|BryanDavis]]
* 03:31 [[etherpad:p/huggle_section]] was archived as [[etherpadbackup:huggle_section]] by [[User:BryanDavis|BryanDavis]]
* 03:30 [[etherpad:p/horizonsso]] was archived as [[etherpadbackup:horizonsso]] by [[User:BryanDavis|BryanDavis]]
* 03:30 [[etherpad:p/hi6ZshFUi2]] was archived as [[etherpadbackup:hi6ZshFUi2]] by [[User:BryanDavis|BryanDavis]]
* 03:29 [[etherpad:p/hhvm]] was archived as [[etherpadbackup:hhvm]] by [[User:BryanDavis|BryanDavis]]
* 03:29 [[etherpad:p/helm-toollabs]] was archived as [[etherpadbackup:helm-toollabs]] by [[User:BryanDavis|BryanDavis]]
* 03:28 [[etherpad:p/helm-chart-development]] was archived as [[etherpadbackup:helm-chart-development]] by [[User:BryanDavis|BryanDavis]]
* 03:28 [[etherpad:p/handoff-wdqs-T388134]] was archived as [[etherpadbackup:handoff-wdqs-T388134]] by [[User:BryanDavis|BryanDavis]]
* 03:27 [[etherpad:p/hackathon-showcase-before-the-catastrophe]] was archived as [[etherpadbackup:hackathon-showcase-before-the-catastrophe]] by [[User:BryanDavis|BryanDavis]]
* 03:27 [[etherpad:p/hackathon-rhinosf1-barriers]] was archived as [[etherpadbackup:hackathon-rhinosf1-barriers]] by [[User:BryanDavis|BryanDavis]]
* 03:26 [[etherpad:p/gridengine-uses]] was archived as [[etherpadbackup:gridengine-uses]] by [[User:BryanDavis|BryanDavis]]
* 03:26 [[etherpad:p/grid-backup]] was archived as [[etherpadbackup:grid-backup]] by [[User:BryanDavis|BryanDavis]]
* 03:25 [[etherpad:p/graphitetodo]] was archived as [[etherpadbackup:graphitetodo]] by [[User:BryanDavis|BryanDavis]]
* 03:25 [[etherpad:p/grantreview]] was archived as [[etherpadbackup:grantreview]] by [[User:BryanDavis|BryanDavis]]
* 03:24 [[etherpad:p/gitlab-sync]] was archived as [[etherpadbackup:gitlab-sync]] by [[User:BryanDavis|BryanDavis]]
* 03:24 [[etherpad:p/git]] was archived as [[etherpadbackup:git]] by [[User:BryanDavis|BryanDavis]]
* 03:23 [[etherpad:p/gerrit-4841]] was archived as [[etherpadbackup:gerrit-4841]] by [[User:BryanDavis|BryanDavis]]
* 03:23 [[etherpad:p/gdash-labels-clarification]] was archived as [[etherpadbackup:gdash-labels-clarification]] by [[User:BryanDavis|BryanDavis]]
* 03:22 [[etherpad:p/gcimail]] was archived as [[etherpadbackup:gcimail]] by [[User:BryanDavis|BryanDavis]]
* 03:22 [[etherpad:p/gannum]] was archived as [[etherpadbackup:gannum]] by [[User:BryanDavis|BryanDavis]]
* 03:21 [[etherpad:p/g5an_WR4fa02lVjmHQse]] was archived as [[etherpadbackup:g5an_WR4fa02lVjmHQse]] by [[User:BryanDavis|BryanDavis]]
* 03:21 [[etherpad:p/g505410announce]] was archived as [[etherpadbackup:g505410announce]] by [[User:BryanDavis|BryanDavis]]
* 03:20 [[etherpad:p/g4-flavors]] was archived as [[etherpadbackup:g4-flavors]] by [[User:BryanDavis|BryanDavis]]
* 03:19 [[etherpad:p/future-of-language-converter]] was archived as [[etherpadbackup:future-of-language-converter]] by [[User:BryanDavis|BryanDavis]]
* 03:19 [[etherpad:p/fullstack_leaks]] was archived as [[etherpadbackup:fullstack_leaks]] by [[User:BryanDavis|BryanDavis]]
* 03:18 [[etherpad:p/fullstack-flapping]] was archived as [[etherpadbackup:fullstack-flapping]] by [[User:BryanDavis|BryanDavis]]
* 03:18 [[etherpad:p/fr-tech_chores_2023_fish_HEAD^]] was archived as [[etherpadbackup:fr-tech_chores_2023_fish_HEAD^]] by [[User:BryanDavis|BryanDavis]]
* 03:17 [[etherpad:p/fp-Hrk0fve_GXMeMZFJk]] was archived as [[etherpadbackup:fp-Hrk0fve_GXMeMZFJk]] by [[User:BryanDavis|BryanDavis]]
* 03:17 [[etherpad:p/floating-ip-aliaser]] was archived as [[etherpadbackup:floating-ip-aliaser]] by [[User:BryanDavis|BryanDavis]]
* 03:16 [[etherpad:p/flagellating-funnel]] was archived as [[etherpadbackup:flagellating-funnel]] by [[User:BryanDavis|BryanDavis]]
* 03:16 [[etherpad:p/extdist]] was archived as [[etherpadbackup:extdist]] by [[User:BryanDavis|BryanDavis]]
* 03:15 [[etherpad:p/existensions]] was archived as [[etherpadbackup:existensions]] by [[User:BryanDavis|BryanDavis]]
* 03:15 [[etherpad:p/eventlogging_stag]] was archived as [[etherpadbackup:eventlogging_stag]] by [[User:BryanDavis|BryanDavis]]
* 03:14 [[etherpad:p/eventbus-statsd]] was archived as [[etherpadbackup:eventbus-statsd]] by [[User:BryanDavis|BryanDavis]]
* 03:14 [[etherpad:p/essex-upgrade]] was archived as [[etherpadbackup:essex-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 03:13 [[etherpad:p/esams-followup]] was archived as [[etherpadbackup:esams-followup]] by [[User:BryanDavis|BryanDavis]]
* 03:13 [[etherpad:p/eqiad1-upgrade-pike]] was archived as [[etherpadbackup:eqiad1-upgrade-pike]] by [[User:BryanDavis|BryanDavis]]
* 03:12 [[etherpad:p/eqiad1-neutron-bootstrap]] was archived as [[etherpadbackup:eqiad1-neutron-bootstrap]] by [[User:BryanDavis|BryanDavis]]
* 03:12 [[etherpad:p/eqiad1-keystone-bootstrap]] was archived as [[etherpadbackup:eqiad1-keystone-bootstrap]] by [[User:BryanDavis|BryanDavis]]
* 03:11 [[etherpad:p/eqiad1]] was archived as [[etherpadbackup:eqiad1]] by [[User:BryanDavis|BryanDavis]]
* 03:11 [[etherpad:p/enwikiSOPADBUpgrades]] was archived as [[etherpadbackup:enwikiSOPADBUpgrades]] by [[User:BryanDavis|BryanDavis]]
* 03:10 [[etherpad:p/engprod-irc]] was archived as [[etherpadbackup:engprod-irc]] by [[User:BryanDavis|BryanDavis]]
* 03:10 [[etherpad:p/empty_vps_projects]] was archived as [[etherpadbackup:empty_vps_projects]] by [[User:BryanDavis|BryanDavis]]
* 03:09 [[etherpad:p/embeddings_and_topic_models]] was archived as [[etherpadbackup:embeddings_and_topic_models]] by [[User:BryanDavis|BryanDavis]]
* 03:09 [[etherpad:p/email-tools]] was archived as [[etherpadbackup:email-tools]] by [[User:BryanDavis|BryanDavis]]
* 03:08 [[etherpad:p/elukey-netflow]] was archived as [[etherpadbackup:elukey-netflow]] by [[User:BryanDavis|BryanDavis]]
* 03:07 [[etherpad:p/elastic-single-node-testing]] was archived as [[etherpadbackup:elastic-single-node-testing]] by [[User:BryanDavis|BryanDavis]]
* 03:07 [[etherpad:p/elastic-2-opensearch-T388610]] was archived as [[etherpadbackup:elastic-2-opensearch-T388610]] by [[User:BryanDavis|BryanDavis]]
* 03:06 [[etherpad:p/el_utils]] was archived as [[etherpadbackup:el_utils]] by [[User:BryanDavis|BryanDavis]]
* 03:06 [[etherpad:p/eiwGl_RxtBIFSyuxv5Yr]] was archived as [[etherpadbackup:eiwGl_RxtBIFSyuxv5Yr]] by [[User:BryanDavis|BryanDavis]]
* 03:05 [[etherpad:p/effiee]] was archived as [[etherpadbackup:effiee]] by [[User:BryanDavis|BryanDavis]]
* 03:05 [[etherpad:p/editing_logging]] was archived as [[etherpadbackup:editing_logging]] by [[User:BryanDavis|BryanDavis]]
* 03:04 [[etherpad:p/edit_types_event_schema]] was archived as [[etherpadbackup:edit_types_event_schema]] by [[User:BryanDavis|BryanDavis]]
* 03:04 [[etherpad:p/edit_history_vetting]] was archived as [[etherpadbackup:edit_history_vetting]] by [[User:BryanDavis|BryanDavis]]
* 03:03 [[etherpad:p/e1]] was archived as [[etherpadbackup:e1]] by [[User:BryanDavis|BryanDavis]]
* 03:03 [[etherpad:p/e]] was archived as [[etherpadbackup:e]] by [[User:BryanDavis|BryanDavis]]
* 03:02 [[etherpad:p/doc-from-git]] was archived as [[etherpadbackup:doc-from-git]] by [[User:BryanDavis|BryanDavis]]
* 03:02 [[etherpad:p/dnsthings]] was archived as [[etherpadbackup:dnsthings]] by [[User:BryanDavis|BryanDavis]]
* 03:01 [[etherpad:p/dns_cleanup]] was archived as [[etherpadbackup:dns_cleanup]] by [[User:BryanDavis|BryanDavis]]
* 03:01 [[etherpad:p/dmitry-onboarding]] was archived as [[etherpadbackup:dmitry-onboarding]] by [[User:BryanDavis|BryanDavis]]
* 03:00 [[etherpad:p/disk_aio_setting]] was archived as [[etherpadbackup:disk_aio_setting]] by [[User:BryanDavis|BryanDavis]]
* 03:00 [[etherpad:p/diff_test]] was archived as [[etherpadbackup:diff_test]] by [[User:BryanDavis|BryanDavis]]
* 02:59 [[etherpad:p/diamond-deployment]] was archived as [[etherpadbackup:diamond-deployment]] by [[User:BryanDavis|BryanDavis]]
* 02:59 [[etherpad:p/dhcp-option82-sunset]] was archived as [[etherpadbackup:dhcp-option82-sunset]] by [[User:BryanDavis|BryanDavis]]
* 02:58 [[etherpad:p/devsummit18-thirdpartymediawiki]] was archived as [[etherpadbackup:devsummit18-thirdpartymediawiki]] by [[User:BryanDavis|BryanDavis]]
* 02:58 [[etherpad:p/devsummit17-support-global-preferences]] was archived as [[etherpadbackup:devsummit17-support-global-preferences]] by [[User:BryanDavis|BryanDavis]]
* 02:57 [[etherpad:p/devsummit17-scaling-database-schema]] was archived as [[etherpadbackup:devsummit17-scaling-database-schema]] by [[User:BryanDavis|BryanDavis]]
* 02:57 [[etherpad:p/devsummit17-integrating-mediawiki]] was archived as [[etherpadbackup:devsummit17-integrating-mediawiki]] by [[User:BryanDavis|BryanDavis]]
* 02:56 [[etherpad:p/devsummit17-algorithmic-dangers]] was archived as [[etherpadbackup:devsummit17-algorithmic-dangers]] by [[User:BryanDavis|BryanDavis]]
* 02:56 [[etherpad:p/devsummit17-ai-wishlist]] was archived as [[etherpadbackup:devsummit17-ai-wishlist]] by [[User:BryanDavis|BryanDavis]]
* 02:55 [[etherpad:p/devsummit17-T149624]] was archived as [[etherpadbackup:devsummit17-T149624]] by [[User:BryanDavis|BryanDavis]]
* 02:55 [[etherpad:p/devsummit17-RedesignSpecialSearchPage]] was archived as [[etherpadbackup:devsummit17-RedesignSpecialSearchPage]] by [[User:BryanDavis|BryanDavis]]
* 02:54 [[etherpad:p/devsummit17-ProjectOREO]] was archived as [[etherpadbackup:devsummit17-ProjectOREO]] by [[User:BryanDavis|BryanDavis]]
* 02:53 [[etherpad:p/devsummit17-CodeReview]] was archived as [[etherpadbackup:devsummit17-CodeReview]] by [[User:BryanDavis|BryanDavis]]
* 02:53 [[etherpad:p/devpowpow-2015-02-25]] was archived as [[etherpadbackup:devpowpow-2015-02-25]] by [[User:BryanDavis|BryanDavis]]
* 02:52 [[etherpad:p/design_review_10_22]] was archived as [[etherpadbackup:design_review_10_22]] by [[User:BryanDavis|BryanDavis]]
* 02:52 [[etherpad:p/design-thingies-jan-android]] was archived as [[etherpadbackup:design-thingies-jan-android]] by [[User:BryanDavis|BryanDavis]]
* 02:51 [[etherpad:p/design]] was archived as [[etherpadbackup:design]] by [[User:BryanDavis|BryanDavis]]
* 02:51 [[etherpad:p/deprecate-precise]] was archived as [[etherpadbackup:deprecate-precise]] by [[User:BryanDavis|BryanDavis]]
* 02:50 [[etherpad:p/depool]] was archived as [[etherpadbackup:depool]] by [[User:BryanDavis|BryanDavis]]
* 02:50 [[etherpad:p/deployworkinggroup]] was archived as [[etherpadbackup:deployworkinggroup]] by [[User:BryanDavis|BryanDavis]]
* 02:49 [[etherpad:p/deploytrainingprep]] was archived as [[etherpadbackup:deploytrainingprep]] by [[User:BryanDavis|BryanDavis]]
* 02:49 [[etherpad:p/deploy-20150427-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150427-SWAT-evening]] by [[User:BryanDavis|BryanDavis]]
* 02:48 [[etherpad:p/deploy-20150416-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150416-SWAT-evening]] by [[User:BryanDavis|BryanDavis]]
* 02:48 [[etherpad:p/deletedesedomains]] was archived as [[etherpadbackup:deletedesedomains]] by [[User:BryanDavis|BryanDavis]]
* 02:47 [[etherpad:p/dear-tool-user]] was archived as [[etherpadbackup:dear-tool-user]] by [[User:BryanDavis|BryanDavis]]
* 02:47 [[etherpad:p/dchanPlate2015]] was archived as [[etherpadbackup:dchanPlate2015]] by [[User:BryanDavis|BryanDavis]]
* 02:46 [[etherpad:p/db-renames]] was archived as [[etherpadbackup:db-renames]] by [[User:BryanDavis|BryanDavis]]
* 02:46 [[etherpad:p/datalossjuly2]] was archived as [[etherpadbackup:datalossjuly2]] by [[User:BryanDavis|BryanDavis]]
* 02:45 [[etherpad:p/cx-presentation]] was archived as [[etherpadbackup:cx-presentation]] by [[User:BryanDavis|BryanDavis]]
* 02:45 [[etherpad:p/cx-markup-alignment-uppercase]] was archived as [[etherpadbackup:cx-markup-alignment-uppercase]] by [[User:BryanDavis|BryanDavis]]
* 02:44 [[etherpad:p/cx-markup-alignment]] was archived as [[etherpadbackup:cx-markup-alignment]] by [[User:BryanDavis|BryanDavis]]
* 02:44 [[etherpad:p/cx-invite]] was archived as [[etherpadbackup:cx-invite]] by [[User:BryanDavis|BryanDavis]]
* 02:43 [[etherpad:p/cx-hack]] was archived as [[etherpadbackup:cx-hack]] by [[User:BryanDavis|BryanDavis]]
* 02:43 [[etherpad:p/cx-composer]] was archived as [[etherpadbackup:cx-composer]] by [[User:BryanDavis|BryanDavis]]
* 02:42 [[etherpad:p/csKD3No9oL]] was archived as [[etherpadbackup:csKD3No9oL]] by [[User:BryanDavis|BryanDavis]]
* 02:42 [[etherpad:p/cronerror]] was archived as [[etherpadbackup:cronerror]] by [[User:BryanDavis|BryanDavis]]
* 02:41 [[etherpad:p/cr2-codfw_fpc0]] was archived as [[etherpadbackup:cr2-codfw_fpc0]] by [[User:BryanDavis|BryanDavis]]
* 02:41 [[etherpad:p/conftool-T379329]] was archived as [[etherpadbackup:conftool-T379329]] by [[User:BryanDavis|BryanDavis]]
* 02:40 [[etherpad:p/completion-ab-sanity-check]] was archived as [[etherpadbackup:completion-ab-sanity-check]] by [[User:BryanDavis|BryanDavis]]
* 02:40 [[etherpad:p/compat-network]] was archived as [[etherpadbackup:compat-network]] by [[User:BryanDavis|BryanDavis]]
* 02:39 [[etherpad:p/commons-ios-fixes]] was archived as [[etherpadbackup:commons-ios-fixes]] by [[User:BryanDavis|BryanDavis]]
* 02:38 [[etherpad:p/commons-cross-wiki-uploads]] was archived as [[etherpadbackup:commons-cross-wiki-uploads]] by [[User:BryanDavis|BryanDavis]]
* 02:38 [[etherpad:p/commons-app-post-draft]] was archived as [[etherpadbackup:commons-app-post-draft]] by [[User:BryanDavis|BryanDavis]]
* 02:37 [[etherpad:p/collation]] was archived as [[etherpadbackup:collation]] by [[User:BryanDavis|BryanDavis]]
* 02:37 [[etherpad:p/codeacademy-diwanship]] was archived as [[etherpadbackup:codeacademy-diwanship]] by [[User:BryanDavis|BryanDavis]]
* 02:36 [[etherpad:p/cna4GMpRlJCm65THgk-l]] was archived as [[etherpadbackup:cna4GMpRlJCm65THgk-l]] by [[User:BryanDavis|BryanDavis]]
* 02:36 [[etherpad:p/cloudvpskernels]] was archived as [[etherpadbackup:cloudvpskernels]] by [[User:BryanDavis|BryanDavis]]
* 02:35 [[etherpad:p/cloudvps-no-agent-forwarding]] was archived as [[etherpadbackup:cloudvps-no-agent-forwarding]] by [[User:BryanDavis|BryanDavis]]
* 02:35 [[etherpad:p/cloudvirt1018-20190213]] was archived as [[etherpadbackup:cloudvirt1018-20190213]] by [[User:BryanDavis|BryanDavis]]
* 02:34 [[etherpad:p/cloudservices1003-upgrade]] was archived as [[etherpadbackup:cloudservices1003-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 02:34 [[etherpad:p/cloudreboots]] was archived as [[etherpadbackup:cloudreboots]] by [[User:BryanDavis|BryanDavis]]
* 02:33 [[etherpad:p/cloudprojectrequeststweak]] was archived as [[etherpadbackup:cloudprojectrequeststweak]] by [[User:BryanDavis|BryanDavis]]
* 02:33 [[etherpad:p/cloudnative]] was archived as [[etherpadbackup:cloudnative]] by [[User:BryanDavis|BryanDavis]]
* 02:32 [[etherpad:p/cloudgw-migration]] was archived as [[etherpadbackup:cloudgw-migration]] by [[User:BryanDavis|BryanDavis]]
* 02:32 [[etherpad:p/cloudfontcdn]] was archived as [[etherpadbackup:cloudfontcdn]] by [[User:BryanDavis|BryanDavis]]
* 02:31 [[etherpad:p/cloudelastic-mystery-red]] was archived as [[etherpadbackup:cloudelastic-mystery-red]] by [[User:BryanDavis|BryanDavis]]
* 02:31 [[etherpad:p/cloudelastic-ipip]] was archived as [[etherpadbackup:cloudelastic-ipip]] by [[User:BryanDavis|BryanDavis]]
* 02:30 [[etherpad:p/cloudelastic-2-opensearch]] was archived as [[etherpadbackup:cloudelastic-2-opensearch]] by [[User:BryanDavis|BryanDavis]]
* 02:30 [[etherpad:p/cloud_purge_2018_ping]] was archived as [[etherpadbackup:cloud_purge_2018_ping]] by [[User:BryanDavis|BryanDavis]]
* 02:29 [[etherpad:p/cloud_announcement_20180910]] was archived as [[etherpadbackup:cloud_announcement_20180910]] by [[User:BryanDavis|BryanDavis]]
* 02:29 [[etherpad:p/cloud-vps-purge-deletions]] was archived as [[etherpadbackup:cloud-vps-purge-deletions]] by [[User:BryanDavis|BryanDavis]]
* 02:28 [[etherpad:p/citoid-TD-fun]] was archived as [[etherpadbackup:citoid-TD-fun]] by [[User:BryanDavis|BryanDavis]]
* 02:28 [[etherpad:p/cirrus_cross_project_search_debug_settings]] was archived as [[etherpadbackup:cirrus_cross_project_search_debug_settings]] by [[User:BryanDavis|BryanDavis]]
* 02:27 [[etherpad:p/ci-apt-docker-why]] was archived as [[etherpadbackup:ci-apt-docker-why]] by [[User:BryanDavis|BryanDavis]]
* 02:27 [[etherpad:p/chrismcmahon]] was archived as [[etherpadbackup:chrismcmahon]] by [[User:BryanDavis|BryanDavis]]
* 02:26 [[etherpad:p/changeuid]] was archived as [[etherpadbackup:changeuid]] by [[User:BryanDavis|BryanDavis]]
* 02:26 [[etherpad:p/changes]] was archived as [[etherpadbackup:changes]] by [[User:BryanDavis|BryanDavis]]
* 02:25 [[etherpad:p/ceph-poc-to-prod]] was archived as [[etherpadbackup:ceph-poc-to-prod]] by [[User:BryanDavis|BryanDavis]]
* 02:25 [[etherpad:p/categories-lag]] was archived as [[etherpadbackup:categories-lag]] by [[User:BryanDavis|BryanDavis]]
* 02:24 [[etherpad:p/calendar-minimums]] was archived as [[etherpadbackup:calendar-minimums]] by [[User:BryanDavis|BryanDavis]]
* 02:24 [[etherpad:p/busterreminder]] was archived as [[etherpadbackup:busterreminder]] by [[User:BryanDavis|BryanDavis]]
* 02:23 [[etherpad:p/bug35939]] was archived as [[etherpadbackup:bug35939]] by [[User:BryanDavis|BryanDavis]]
* 02:22 [[etherpad:p/brokenpms]] was archived as [[etherpadbackup:brokenpms]] by [[User:BryanDavis|BryanDavis]]
* 02:22 [[etherpad:p/bloated-website-code-drains-yo]] was archived as [[etherpadbackup:bloated-website-code-drains-yo]] by [[User:BryanDavis|BryanDavis]]
* 02:21 [[etherpad:p/bios-be-gone]] was archived as [[etherpadbackup:bios-be-gone]] by [[User:BryanDavis|BryanDavis]]
* 02:21 [[etherpad:p/bigbrother-almost-no-more]] was archived as [[etherpadbackup:bigbrother-almost-no-more]] by [[User:BryanDavis|BryanDavis]]
* 02:20 [[etherpad:p/beta-new-ext]] was archived as [[etherpadbackup:beta-new-ext]] by [[User:BryanDavis|BryanDavis]]
* 02:20 [[etherpad:p/bernd-onboarding]] was archived as [[etherpadbackup:bernd-onboarding]] by [[User:BryanDavis|BryanDavis]]
* 02:19 [[etherpad:p/beginner-hackathons-thoughts]] was archived as [[etherpadbackup:beginner-hackathons-thoughts]] by [[User:BryanDavis|BryanDavis]]
* 02:19 [[etherpad:p/bash]] was archived as [[etherpadbackup:bash]] by [[User:BryanDavis|BryanDavis]]
* 02:18 [[etherpad:p/bangarang]] was archived as [[etherpadbackup:bangarang]] by [[User:BryanDavis|BryanDavis]]
* 02:18 [[etherpad:p/band_names_web]] was archived as [[etherpadbackup:band_names_web]] by [[User:BryanDavis|BryanDavis]]
* 02:17 [[etherpad:p/avro]] was archived as [[etherpadbackup:avro]] by [[User:BryanDavis|BryanDavis]]
* 02:17 [[etherpad:p/aqs-standup]] was archived as [[etherpadbackup:aqs-standup]] by [[User:BryanDavis|BryanDavis]]
* 02:16 [[etherpad:p/app_q3_planning]] was archived as [[etherpadbackup:app_q3_planning]] by [[User:BryanDavis|BryanDavis]]
* 02:16 [[etherpad:p/app_q3_health_check]] was archived as [[etherpadbackup:app_q3_health_check]] by [[User:BryanDavis|BryanDavis]]
* 02:15 [[etherpad:p/app_process_improvements]] was archived as [[etherpadbackup:app_process_improvements]] by [[User:BryanDavis|BryanDavis]]
* 02:15 [[etherpad:p/appQ4planning]] was archived as [[etherpadbackup:appQ4planning]] by [[User:BryanDavis|BryanDavis]]
* 02:14 [[etherpad:p/app-strategy-discussion]] was archived as [[etherpadbackup:app-strategy-discussion]] by [[User:BryanDavis|BryanDavis]]
* 02:14 [[etherpad:p/app-server-upgrade]] was archived as [[etherpadbackup:app-server-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 02:13 [[etherpad:p/app-1-1beta4-3-1beta3-release-notes]] was archived as [[etherpadbackup:app-1-1beta4-3-1beta3-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:13 [[etherpad:p/apiv2]] was archived as [[etherpadbackup:apiv2]] by [[User:BryanDavis|BryanDavis]]
* 02:12 [[etherpad:p/api.php]] was archived as [[etherpadbackup:api.php]] by [[User:BryanDavis|BryanDavis]]
* 02:12 [[etherpad:p/androidappfeedback]] was archived as [[etherpadbackup:androidappfeedback]] by [[User:BryanDavis|BryanDavis]]
* 02:11 [[etherpad:p/android-ime]] was archived as [[etherpadbackup:android-ime]] by [[User:BryanDavis|BryanDavis]]
* 02:11 [[etherpad:p/android-commons-app-1-0]] was archived as [[etherpadbackup:android-commons-app-1-0]] by [[User:BryanDavis|BryanDavis]]
* 02:10 [[etherpad:p/android-1-1rc1-release-notes]] was archived as [[etherpadbackup:android-1-1rc1-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:10 [[etherpad:p/android-1-1beta1-release-notes]] was archived as [[etherpadbackup:android-1-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:09 [[etherpad:p/android-1-1-iOS-3-1-third-beta-notes]] was archived as [[etherpadbackup:android-1-1-iOS-3-1-third-beta-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:08 [[etherpad:p/android-1-1-beta2-release-notes]] was archived as [[etherpadbackup:android-1-1-beta2-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:08 [[etherpad:p/analytics163]] was archived as [[etherpadbackup:analytics163]] by [[User:BryanDavis|BryanDavis]]
* 02:07 [[etherpad:p/analytics1003-reinstall]] was archived as [[etherpadbackup:analytics1003-reinstall]] by [[User:BryanDavis|BryanDavis]]
* 02:07 [[etherpad:p/analytics-todo]] was archived as [[etherpadbackup:analytics-todo]] by [[User:BryanDavis|BryanDavis]]
* 02:06 [[etherpad:p/analytics-tech-debt]] was archived as [[etherpadbackup:analytics-tech-debt]] by [[User:BryanDavis|BryanDavis]]
* 02:06 [[etherpad:p/analytics-staff-meeting]] was archived as [[etherpadbackup:analytics-staff-meeting]] by [[User:BryanDavis|BryanDavis]]
* 02:05 [[etherpad:p/analytics-row-d-maintenance]] was archived as [[etherpadbackup:analytics-row-d-maintenance]] by [[User:BryanDavis|BryanDavis]]
* 02:05 [[etherpad:p/analytics-retrospective]] was archived as [[etherpadbackup:analytics-retrospective]] by [[User:BryanDavis|BryanDavis]]
* 02:04 [[etherpad:p/analytics-ops-druid]] was archived as [[etherpadbackup:analytics-ops-druid]] by [[User:BryanDavis|BryanDavis]]
* 02:04 [[etherpad:p/analytics-ops]] was archived as [[etherpadbackup:analytics-ops]] by [[User:BryanDavis|BryanDavis]]
* 02:03 [[etherpad:p/analytics-oozie-13102016]] was archived as [[etherpadbackup:analytics-oozie-13102016]] by [[User:BryanDavis|BryanDavis]]
* 02:03 [[etherpad:p/analytics-naming]] was archived as [[etherpadbackup:analytics-naming]] by [[User:BryanDavis|BryanDavis]]
* 02:02 [[etherpad:p/analytics-mwds-2015]] was archived as [[etherpadbackup:analytics-mwds-2015]] by [[User:BryanDavis|BryanDavis]]
* 02:02 [[etherpad:p/analytics-ingestion]] was archived as [[etherpadbackup:analytics-ingestion]] by [[User:BryanDavis|BryanDavis]]
* 02:01 [[etherpad:p/analytics-hive-udf]] was archived as [[etherpadbackup:analytics-hive-udf]] by [[User:BryanDavis|BryanDavis]]
* 02:01 [[etherpad:p/analytics-gobblin-mess]] was archived as [[etherpadbackup:analytics-gobblin-mess]] by [[User:BryanDavis|BryanDavis]]
* 02:00 [[etherpad:p/analytics-goals]] was archived as [[etherpadbackup:analytics-goals]] by [[User:BryanDavis|BryanDavis]]
* 02:00 [[etherpad:p/analytics-eventlogging-postmortem-2014-11]] was archived as [[etherpadbackup:analytics-eventlogging-postmortem-2014-11]] by [[User:BryanDavis|BryanDavis]]
* 01:59 [[etherpad:p/analytics-email-drafts]] was archived as [[etherpadbackup:analytics-email-drafts]] by [[User:BryanDavis|BryanDavis]]
* 01:59 [[etherpad:p/analytics-druid-upgrade]] was archived as [[etherpadbackup:analytics-druid-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:58 [[etherpad:p/analytics-druid-migration]] was archived as [[etherpadbackup:analytics-druid-migration]] by [[User:BryanDavis|BryanDavis]]
* 01:58 [[etherpad:p/analytics-deploy-aqs]] was archived as [[etherpadbackup:analytics-deploy-aqs]] by [[User:BryanDavis|BryanDavis]]
* 01:57 [[etherpad:p/analytics-bikeshedding]] was archived as [[etherpadbackup:analytics-bikeshedding]] by [[User:BryanDavis|BryanDavis]]
* 01:57 [[etherpad:p/analytics-bikeshed]] was archived as [[etherpadbackup:analytics-bikeshed]] by [[User:BryanDavis|BryanDavis]]
* 01:56 [[etherpad:p/analytics-analytics1003-jessie-upgrade]] was archived as [[etherpadbackup:analytics-analytics1003-jessie-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:56 [[etherpad:p/analytics-VE]] was archived as [[etherpadbackup:analytics-VE]] by [[User:BryanDavis|BryanDavis]]
* 01:55 [[etherpad:p/analytics-73331]] was archived as [[etherpadbackup:analytics-73331]] by [[User:BryanDavis|BryanDavis]]
* 01:55 [[etherpad:p/analytics-72745]] was archived as [[etherpadbackup:analytics-72745]] by [[User:BryanDavis|BryanDavis]]
* 01:54 [[etherpad:p/analytics-72740]] was archived as [[etherpadbackup:analytics-72740]] by [[User:BryanDavis|BryanDavis]]
* 01:54 [[etherpad:p/analytics-72739]] was archived as [[etherpadbackup:analytics-72739]] by [[User:BryanDavis|BryanDavis]]
* 01:53 [[etherpad:p/analytics-72738]] was archived as [[etherpadbackup:analytics-72738]] by [[User:BryanDavis|BryanDavis]]
* 01:52 [[etherpad:p/analytics-72737]] was archived as [[etherpadbackup:analytics-72737]] by [[User:BryanDavis|BryanDavis]]
* 01:52 [[etherpad:p/analytics-72736]] was archived as [[etherpadbackup:analytics-72736]] by [[User:BryanDavis|BryanDavis]]
* 01:51 [[etherpad:p/analytics-72735]] was archived as [[etherpadbackup:analytics-72735]] by [[User:BryanDavis|BryanDavis]]
* 01:51 [[etherpad:p/analytics-69254]] was archived as [[etherpadbackup:analytics-69254]] by [[User:BryanDavis|BryanDavis]]
* 01:50 [[etherpad:p/alpha-signature-change]] was archived as [[etherpadbackup:alpha-signature-change]] by [[User:BryanDavis|BryanDavis]]
* 01:50 [[etherpad:p/ai_vision_wikimedia]] was archived as [[etherpadbackup:ai_vision_wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 01:49 [[etherpad:p/aharoni-20160420]] was archived as [[etherpadbackup:aharoni-20160420]] by [[User:BryanDavis|BryanDavis]]
* 01:49 [[etherpad:p/aftereffects]] was archived as [[etherpadbackup:aftereffects]] by [[User:BryanDavis|BryanDavis]]
* 01:48 [[etherpad:p/admin_accounts_cleanup]] was archived as [[etherpadbackup:admin_accounts_cleanup]] by [[User:BryanDavis|BryanDavis]]
* 01:48 [[etherpad:p/admin]] was archived as [[etherpadbackup:admin]] by [[User:BryanDavis|BryanDavis]]
* 01:47 [[etherpad:p/addwiki]] was archived as [[etherpadbackup:addwiki]] by [[User:BryanDavis|BryanDavis]]
* 01:47 [[etherpad:p/addshore]] was archived as [[etherpadbackup:addshore]] by [[User:BryanDavis|BryanDavis]]
* 01:46 [[etherpad:p/actionbar_checklist]] was archived as [[etherpadbackup:actionbar_checklist]] by [[User:BryanDavis|BryanDavis]]
* 01:46 [[etherpad:p/access_request_wikis]] was archived as [[etherpadbackup:access_request_wikis]] by [[User:BryanDavis|BryanDavis]]
* 01:45 [[etherpad:p/absents]] was archived as [[etherpadbackup:absents]] by [[User:BryanDavis|BryanDavis]]
* 01:45 [[etherpad:p/a4NCijsjQp]] was archived as [[etherpadbackup:a4NCijsjQp]] by [[User:BryanDavis|BryanDavis]]
* 01:44 [[etherpad:p/ZVFiVE3E09]] was archived as [[etherpadbackup:ZVFiVE3E09]] by [[User:BryanDavis|BryanDavis]]
* 01:44 [[etherpad:p/XcC5u59nDdRc_itxINnb]] was archived as [[etherpadbackup:XcC5u59nDdRc_itxINnb]] by [[User:BryanDavis|BryanDavis]]
* 01:43 [[etherpad:p/XH6WkSmARmw8dy7UMiIR]] was archived as [[etherpadbackup:XH6WkSmARmw8dy7UMiIR]] by [[User:BryanDavis|BryanDavis]]
* 01:43 [[etherpad:p/WsWWUiRllf]] was archived as [[etherpadbackup:WsWWUiRllf]] by [[User:BryanDavis|BryanDavis]]
* 01:42 [[etherpad:p/WorT2b8qpR]] was archived as [[etherpadbackup:WorT2b8qpR]] by [[User:BryanDavis|BryanDavis]]
* 01:42 [[etherpad:p/Wj4HcOWoHk]] was archived as [[etherpadbackup:Wj4HcOWoHk]] by [[User:BryanDavis|BryanDavis]]
* 01:41 [[etherpad:p/WikipediaiOSTesting]] was archived as [[etherpadbackup:WikipediaiOSTesting]] by [[User:BryanDavis|BryanDavis]]
* 01:41 [[etherpad:p/WikipediaZeroRetrospective]] was archived as [[etherpadbackup:WikipediaZeroRetrospective]] by [[User:BryanDavis|BryanDavis]]
* 01:40 [[etherpad:p/WikipediaPhoneGapAndroidMeetup]] was archived as [[etherpadbackup:WikipediaPhoneGapAndroidMeetup]] by [[User:BryanDavis|BryanDavis]]
* 01:40 [[etherpad:p/WikipediaMobileiOS3-1Changelog]] was archived as [[etherpadbackup:WikipediaMobileiOS3-1Changelog]] by [[User:BryanDavis|BryanDavis]]
* 01:39 [[etherpad:p/WikipediaMobileAndroidV1-0-1]] was archived as [[etherpadbackup:WikipediaMobileAndroidV1-0-1]] by [[User:BryanDavis|BryanDavis]]
* 01:39 [[etherpad:p/WikipediaMobileAndroidRelease]] was archived as [[etherpadbackup:WikipediaMobileAndroidRelease]] by [[User:BryanDavis|BryanDavis]]
* 01:38 [[etherpad:p/WikipediaMobileAndroid-V1-0-2]] was archived as [[etherpadbackup:WikipediaMobileAndroid-V1-0-2]] by [[User:BryanDavis|BryanDavis]]
* 01:37 [[etherpad:p/WikipediaMobile-3-1-1-scratchpad]] was archived as [[etherpadbackup:WikipediaMobile-3-1-1-scratchpad]] by [[User:BryanDavis|BryanDavis]]
* 01:37 [[etherpad:p/WikipediaLite]] was archived as [[etherpadbackup:WikipediaLite]] by [[User:BryanDavis|BryanDavis]]
* 01:36 [[etherpad:p/Wikipedia-iOS-Rejection-Message]] was archived as [[etherpadbackup:Wikipedia-iOS-Rejection-Message]] by [[User:BryanDavis|BryanDavis]]
* 01:36 [[etherpad:p/Wikipedia-iOS-3-1-1-issues]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1-issues]] by [[User:BryanDavis|BryanDavis]]
* 01:35 [[etherpad:p/Wikipedia-iOS-3-1-1]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1]] by [[User:BryanDavis|BryanDavis]]
* 01:35 [[etherpad:p/Wikipedia-iOS]] was archived as [[etherpadbackup:Wikipedia-iOS]] by [[User:BryanDavis|BryanDavis]]
* 01:34 [[etherpad:p/WikipdiaMobileRC4]] was archived as [[etherpadbackup:WikipdiaMobileRC4]] by [[User:BryanDavis|BryanDavis]]
* 01:34 [[etherpad:p/WikimediaTelepresence]] was archived as [[etherpadbackup:WikimediaTelepresence]] by [[User:BryanDavis|BryanDavis]]
* 01:33 [[etherpad:p/Wikimania_Hackathon_2025_-_Opening_Ceremony]] was archived as [[etherpadbackup:Wikimania_Hackathon_2025_-_Opening_Ceremony]] by [[User:BryanDavis|BryanDavis]]
* 01:33 [[etherpad:p/Wikimania_2023_Hackathon]] was archived as [[etherpadbackup:Wikimania_2023_Hackathon]] by [[User:BryanDavis|BryanDavis]]
* 01:32 [[etherpad:p/Wikigrok_Q4]] was archived as [[etherpadbackup:Wikigrok_Q4]] by [[User:BryanDavis|BryanDavis]]
* 01:32 [[etherpad:p/Wikidata_technical_needs]] was archived as [[etherpadbackup:Wikidata_technical_needs]] by [[User:BryanDavis|BryanDavis]]
* 01:31 [[etherpad:p/WikiGrokTest1]] was archived as [[etherpadbackup:WikiGrokTest1]] by [[User:BryanDavis|BryanDavis]]
* 01:31 [[etherpad:p/WikiDev16-T114045]] was archived as [[etherpadbackup:WikiDev16-T114045]] by [[User:BryanDavis|BryanDavis]]
* 01:30 [[etherpad:p/Watchlist-20beta-20status]] was archived as [[etherpadbackup:Watchlist-20beta-20status]] by [[User:BryanDavis|BryanDavis]]
* 01:30 [[etherpad:p/WRN202003]] was archived as [[etherpadbackup:WRN202003]] by [[User:BryanDavis|BryanDavis]]
* 01:29 [[etherpad:p/WRN201409]] was archived as [[etherpadbackup:WRN201409]] by [[User:BryanDavis|BryanDavis]]
* 01:29 [[etherpad:p/WRN201408]] was archived as [[etherpadbackup:WRN201408]] by [[User:BryanDavis|BryanDavis]]
* 01:28 [[etherpad:p/WP_BRD_generally]] was archived as [[etherpadbackup:WP_BRD_generally]] by [[User:BryanDavis|BryanDavis]]
* 01:28 [[etherpad:p/WMTC19-T238265]] was archived as [[etherpadbackup:WMTC19-T238265]] by [[User:BryanDavis|BryanDavis]]
* 01:27 [[etherpad:p/WMTC19-T238227]] was archived as [[etherpadbackup:WMTC19-T238227]] by [[User:BryanDavis|BryanDavis]]
* 01:27 [[etherpad:p/WMTC19-T234662]] was archived as [[etherpadbackup:WMTC19-T234662]] by [[User:BryanDavis|BryanDavis]]
* 01:26 [[etherpad:p/WMTC19-T234655]] was archived as [[etherpadbackup:WMTC19-T234655]] by [[User:BryanDavis|BryanDavis]]
* 01:26 [[etherpad:p/WMTC19-T234654]] was archived as [[etherpadbackup:WMTC19-T234654]] by [[User:BryanDavis|BryanDavis]]
* 01:25 [[etherpad:p/WMTC19-T234649]] was archived as [[etherpadbackup:WMTC19-T234649]] by [[User:BryanDavis|BryanDavis]]
* 01:25 [[etherpad:p/WMTC19-T234641]] was archived as [[etherpadbackup:WMTC19-T234641]] by [[User:BryanDavis|BryanDavis]]
* 01:24 [[etherpad:p/WMTC19-T234636]] was archived as [[etherpadbackup:WMTC19-T234636]] by [[User:BryanDavis|BryanDavis]]
* 01:24 [[etherpad:p/WMTC19-T234632]] was archived as [[etherpadbackup:WMTC19-T234632]] by [[User:BryanDavis|BryanDavis]]
* 01:23 [[etherpad:p/WMHack_Feature_Ideas_for_WorkAdventure]] was archived as [[etherpadbackup:WMHack_Feature_Ideas_for_WorkAdventure]] by [[User:BryanDavis|BryanDavis]]
* 01:23 [[etherpad:p/WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] was archived as [[etherpadbackup:WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] by [[User:BryanDavis|BryanDavis]]
* 01:22 [[etherpad:p/WMF_Research_Office_Hours|Notes]] was archived as [[etherpadbackup:WMF_Research_Office_Hours|Notes]] by [[User:BryanDavis|BryanDavis]]
* 01:21 [[etherpad:p/WMCS-techsupport-task]] was archived as [[etherpadbackup:WMCS-techsupport-task]] by [[User:BryanDavis|BryanDavis]]
* 01:21 [[etherpad:p/WMCS-infrafoundations-2021-11-24]] was archived as [[etherpadbackup:WMCS-infrafoundations-2021-11-24]] by [[User:BryanDavis|BryanDavis]]
* 01:20 [[etherpad:p/WMCS-Renaming-Announce]] was archived as [[etherpadbackup:WMCS-Renaming-Announce]] by [[User:BryanDavis|BryanDavis]]
* 01:20 [[etherpad:p/WMCS-PDU-ops-2019-10-24]] was archived as [[etherpadbackup:WMCS-PDU-ops-2019-10-24]] by [[User:BryanDavis|BryanDavis]]
* 01:19 [[etherpad:p/WMCS-2026-02-12]] was archived as [[etherpadbackup:WMCS-2026-02-12]] by [[User:BryanDavis|BryanDavis]]
* 01:19 [[etherpad:p/WMCS-2025-12-18]] was archived as [[etherpadbackup:WMCS-2025-12-18]] by [[User:BryanDavis|BryanDavis]]
* 01:18 [[etherpad:p/WMCS-2025-09-04]] was archived as [[etherpadbackup:WMCS-2025-09-04]] by [[User:BryanDavis|BryanDavis]]
* 01:18 [[etherpad:p/WMCS-2025-08-14]] was archived as [[etherpadbackup:WMCS-2025-08-14]] by [[User:BryanDavis|BryanDavis]]
* 01:17 [[etherpad:p/WMCS-2024-02-07]] was archived as [[etherpadbackup:WMCS-2024-02-07]] by [[User:BryanDavis|BryanDavis]]
* 01:17 [[etherpad:p/WMCS-2023-10-04]] was archived as [[etherpadbackup:WMCS-2023-10-04]] by [[User:BryanDavis|BryanDavis]]
* 01:16 [[etherpad:p/WMCS-2023-09-20]] was archived as [[etherpadbackup:WMCS-2023-09-20]] by [[User:BryanDavis|BryanDavis]]
* 01:16 [[etherpad:p/WMCS-2023-08-10]] was archived as [[etherpadbackup:WMCS-2023-08-10]] by [[User:BryanDavis|BryanDavis]]
* 01:15 [[etherpad:p/WMCS-2023-06-28]] was archived as [[etherpadbackup:WMCS-2023-06-28]] by [[User:BryanDavis|BryanDavis]]
* 01:15 [[etherpad:p/WMCS-2023-04-09-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2023-04-09-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:14 [[etherpad:p/WMCS-2022-11-09]] was archived as [[etherpadbackup:WMCS-2022-11-09]] by [[User:BryanDavis|BryanDavis]]
* 01:14 [[etherpad:p/WMCS-2022-09-07]] was archived as [[etherpadbackup:WMCS-2022-09-07]] by [[User:BryanDavis|BryanDavis]]
* 01:13 [[etherpad:p/WMCS-2022-07-20]] was archived as [[etherpadbackup:WMCS-2022-07-20]] by [[User:BryanDavis|BryanDavis]]
* 01:13 [[etherpad:p/WMCS-2022-07-13]] was archived as [[etherpadbackup:WMCS-2022-07-13]] by [[User:BryanDavis|BryanDavis]]
* 01:12 [[etherpad:p/WMCS-2022-05-11-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2022-05-11-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:12 [[etherpad:p/WMCS-2021-12-15]] was archived as [[etherpadbackup:WMCS-2021-12-15]] by [[User:BryanDavis|BryanDavis]]
* 01:11 [[etherpad:p/WMCS-2021-11-03]] was archived as [[etherpadbackup:WMCS-2021-11-03]] by [[User:BryanDavis|BryanDavis]]
* 01:10 [[etherpad:p/WMCS-2020-12-02]] was archived as [[etherpadbackup:WMCS-2020-12-02]] by [[User:BryanDavis|BryanDavis]]
* 01:10 [[etherpad:p/WMCS-2020-11-25]] was archived as [[etherpadbackup:WMCS-2020-11-25]] by [[User:BryanDavis|BryanDavis]]
* 01:09 [[etherpad:p/WMCS-2020-11-04]] was archived as [[etherpadbackup:WMCS-2020-11-04]] by [[User:BryanDavis|BryanDavis]]
* 01:09 [[etherpad:p/WMCS-2020-04-22]] was archived as [[etherpadbackup:WMCS-2020-04-22]] by [[User:BryanDavis|BryanDavis]]
* 01:08 [[etherpad:p/WMCS-2020-01-07]] was archived as [[etherpadbackup:WMCS-2020-01-07]] by [[User:BryanDavis|BryanDavis]]
* 01:08 [[etherpad:p/WMCS-2019-10-24]] was archived as [[etherpadbackup:WMCS-2019-10-24]] by [[User:BryanDavis|BryanDavis]]
* 01:07 [[etherpad:p/WMCS-2019-09-03]] was archived as [[etherpadbackup:WMCS-2019-09-03]] by [[User:BryanDavis|BryanDavis]]
* 01:07 [[etherpad:p/WMCS-2019-07-30]] was archived as [[etherpadbackup:WMCS-2019-07-30]] by [[User:BryanDavis|BryanDavis]]
* 01:06 [[etherpad:p/WMCS-2019-01-29]] was archived as [[etherpadbackup:WMCS-2019-01-29]] by [[User:BryanDavis|BryanDavis]]
* 01:06 [[etherpad:p/WMCS-2018-10-02]] was archived as [[etherpadbackup:WMCS-2018-10-02]] by [[User:BryanDavis|BryanDavis]]
* 01:05 [[etherpad:p/WMCS-2018-08-14]] was archived as [[etherpadbackup:WMCS-2018-08-14]] by [[User:BryanDavis|BryanDavis]]
* 01:05 [[etherpad:p/WMCS-2018-07-31]] was archived as [[etherpadbackup:WMCS-2018-07-31]] by [[User:BryanDavis|BryanDavis]]
* 01:04 [[etherpad:p/WMCS-2018-07-10]] was archived as [[etherpadbackup:WMCS-2018-07-10]] by [[User:BryanDavis|BryanDavis]]
* 01:04 [[etherpad:p/WMCS-2018-06-19]] was archived as [[etherpadbackup:WMCS-2018-06-19]] by [[User:BryanDavis|BryanDavis]]
* 01:03 [[etherpad:p/WMCS-2018-06-05]] was archived as [[etherpadbackup:WMCS-2018-06-05]] by [[User:BryanDavis|BryanDavis]]
* 01:03 [[etherpad:p/WMCS-2018-05-29]] was archived as [[etherpadbackup:WMCS-2018-05-29]] by [[User:BryanDavis|BryanDavis]]
* 01:02 [[etherpad:p/WMCS-2018-05-01]] was archived as [[etherpadbackup:WMCS-2018-05-01]] by [[User:BryanDavis|BryanDavis]]
* 01:02 [[etherpad:p/WMCS-2018-04-10]] was archived as [[etherpadbackup:WMCS-2018-04-10]] by [[User:BryanDavis|BryanDavis]]
* 01:01 [[etherpad:p/WMCS-2018-04-03]] was archived as [[etherpadbackup:WMCS-2018-04-03]] by [[User:BryanDavis|BryanDavis]]
* 01:01 [[etherpad:p/WMCS-2018-02-13]] was archived as [[etherpadbackup:WMCS-2018-02-13]] by [[User:BryanDavis|BryanDavis]]
* 01:00 [[etherpad:p/WMCS-2018-02-05]] was archived as [[etherpadbackup:WMCS-2018-02-05]] by [[User:BryanDavis|BryanDavis]]
* 01:00 [[etherpad:p/WMCS-2018-01-23]] was archived as [[etherpadbackup:WMCS-2018-01-23]] by [[User:BryanDavis|BryanDavis]]
* 00:59 [[etherpad:p/WMCS-2018-01-09]] was archived as [[etherpadbackup:WMCS-2018-01-09]] by [[User:BryanDavis|BryanDavis]]
* 00:59 [[etherpad:p/WMCS-2017-12-19]] was archived as [[etherpadbackup:WMCS-2017-12-19]] by [[User:BryanDavis|BryanDavis]]
* 00:58 [[etherpad:p/WMCS-2017-11-28]] was archived as [[etherpadbackup:WMCS-2017-11-28]] by [[User:BryanDavis|BryanDavis]]
* 00:58 [[etherpad:p/WMCS-2017-11-21]] was archived as [[etherpadbackup:WMCS-2017-11-21]] by [[User:BryanDavis|BryanDavis]]
* 00:57 [[etherpad:p/WMCS-2017-11-14]] was archived as [[etherpadbackup:WMCS-2017-11-14]] by [[User:BryanDavis|BryanDavis]]
* 00:57 [[etherpad:p/WMCS-2017-11-07]] was archived as [[etherpadbackup:WMCS-2017-11-07]] by [[User:BryanDavis|BryanDavis]]
* 00:56 [[etherpad:p/WMCS-2017-10-30]] was archived as [[etherpadbackup:WMCS-2017-10-30]] by [[User:BryanDavis|BryanDavis]]
* 00:56 [[etherpad:p/WMCS-2017-10-24]] was archived as [[etherpadbackup:WMCS-2017-10-24]] by [[User:BryanDavis|BryanDavis]]
* 00:55 [[etherpad:p/WMCS-2017-10-03]] was archived as [[etherpadbackup:WMCS-2017-10-03]] by [[User:BryanDavis|BryanDavis]]
* 00:54 [[etherpad:p/WMCS-2017-09-19]] was archived as [[etherpadbackup:WMCS-2017-09-19]] by [[User:BryanDavis|BryanDavis]]
* 00:54 [[etherpad:p/WMCS-2017-05-15]] was archived as [[etherpadbackup:WMCS-2017-05-15]] by [[User:BryanDavis|BryanDavis]]
* 00:53 [[etherpad:p/WMCS-2017-05-08]] was archived as [[etherpadbackup:WMCS-2017-05-08]] by [[User:BryanDavis|BryanDavis]]
* 00:53 [[etherpad:p/WMCS-2017-04-17]] was archived as [[etherpadbackup:WMCS-2017-04-17]] by [[User:BryanDavis|BryanDavis]]
* 00:52 [[etherpad:p/WMCS-2017-04-10]] was archived as [[etherpadbackup:WMCS-2017-04-10]] by [[User:BryanDavis|BryanDavis]]
* 00:52 [[etherpad:p/WM2024_Day4_Lviv-_Rooms_21+22+23]] was archived as [[etherpadbackup:WM2024_Day4_Lviv-_Rooms_21+22+23]] by [[User:BryanDavis|BryanDavis]]
* 00:51 [[etherpad:p/WM2024_Day1_Warsaw-_Rooms_20+24]] was archived as [[etherpadbackup:WM2024_Day1_Warsaw-_Rooms_20+24]] by [[User:BryanDavis|BryanDavis]]
* 00:51 [[etherpad:p/WLMshowcase]] was archived as [[etherpadbackup:WLMshowcase]] by [[User:BryanDavis|BryanDavis]]
* 00:50 [[etherpad:p/WLMShowcase2]] was archived as [[etherpadbackup:WLMShowcase2]] by [[User:BryanDavis|BryanDavis]]
* 00:50 [[etherpad:p/WLMMobile]] was archived as [[etherpadbackup:WLMMobile]] by [[User:BryanDavis|BryanDavis]]
* 00:49 [[etherpad:p/WLMMarketToDo]] was archived as [[etherpadbackup:WLMMarketToDo]] by [[User:BryanDavis|BryanDavis]]
* 00:49 [[etherpad:p/WLMAppEmail]] was archived as [[etherpadbackup:WLMAppEmail]] by [[User:BryanDavis|BryanDavis]]
* 00:48 [[etherpad:p/WLM-on-cluster]] was archived as [[etherpadbackup:WLM-on-cluster]] by [[User:BryanDavis|BryanDavis]]
* 00:48 [[etherpad:p/WLM-mobile]] was archived as [[etherpadbackup:WLM-mobile]] by [[User:BryanDavis|BryanDavis]]
* 00:47 [[etherpad:p/WLM-API-to-WMF]] was archived as [[etherpadbackup:WLM-API-to-WMF]] by [[User:BryanDavis|BryanDavis]]
* 00:47 [[etherpad:p/WLM-2013-07-05]] was archived as [[etherpadbackup:WLM-2013-07-05]] by [[User:BryanDavis|BryanDavis]]
* 00:46 [[etherpad:p/VzkZSy0PKK]] was archived as [[etherpadbackup:VzkZSy0PKK]] by [[User:BryanDavis|BryanDavis]]
* 00:46 [[etherpad:p/VisualEditorTeamLunchOrder20140327]] was archived as [[etherpadbackup:VisualEditorTeamLunchOrder20140327]] by [[User:BryanDavis|BryanDavis]]
* 00:45 [[etherpad:p/VelocityNYC2015-Day1]] was archived as [[etherpadbackup:VelocityNYC2015-Day1]] by [[User:BryanDavis|BryanDavis]]
* 00:45 [[etherpad:p/Vagrant-wannabees]] was archived as [[etherpadbackup:Vagrant-wannabees]] by [[User:BryanDavis|BryanDavis]]
* 00:44 [[etherpad:p/VPT]] was archived as [[etherpadbackup:VPT]] by [[User:BryanDavis|BryanDavis]]
* 00:44 [[etherpad:p/VM_puppet_failures]] was archived as [[etherpadbackup:VM_puppet_failures]] by [[User:BryanDavis|BryanDavis]]
* 00:43 [[etherpad:p/VM_puppet_disabled]] was archived as [[etherpadbackup:VM_puppet_disabled]] by [[User:BryanDavis|BryanDavis]]
* 00:43 [[etherpad:p/VE_bugs]] was archived as [[etherpadbackup:VE_bugs]] by [[User:BryanDavis|BryanDavis]]
* 00:42 [[etherpad:p/VETOC]] was archived as [[etherpadbackup:VETOC]] by [[User:BryanDavis|BryanDavis]]
* 00:42 [[etherpad:p/VEQ3Perf]] was archived as [[etherpadbackup:VEQ3Perf]] by [[User:BryanDavis|BryanDavis]]
* 00:41 [[etherpad:p/VE-Chromium-Upstream]] was archived as [[etherpadbackup:VE-Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]]
* 00:41 [[etherpad:p/UwSprintAugust2012]] was archived as [[etherpadbackup:UwSprintAugust2012]] by [[User:BryanDavis|BryanDavis]]
* 00:40 [[etherpad:p/Upload_debugging]] was archived as [[etherpadbackup:Upload_debugging]] by [[User:BryanDavis|BryanDavis]]
* 00:39 [[etherpad:p/UploadWizard-and-EventLogging]] was archived as [[etherpadbackup:UploadWizard-and-EventLogging]] by [[User:BryanDavis|BryanDavis]]
* 00:39 [[etherpad:p/Upgrade-now-bitches]] was archived as [[etherpadbackup:Upgrade-now-bitches]] by [[User:BryanDavis|BryanDavis]]
* 00:38 [[etherpad:p/UniqueServiceGroups]] was archived as [[etherpadbackup:UniqueServiceGroups]] by [[User:BryanDavis|BryanDavis]]
* 00:38 [[etherpad:p/Udeme_Emmanson]] was archived as [[etherpadbackup:Udeme_Emmanson]] by [[User:BryanDavis|BryanDavis]]
* 00:37 [[etherpad:p/UShQLRyUMv]] was archived as [[etherpadbackup:UShQLRyUMv]] by [[User:BryanDavis|BryanDavis]]
* 00:37 [[etherpad:p/UA-SchemaUpdate]] was archived as [[etherpadbackup:UA-SchemaUpdate]] by [[User:BryanDavis|BryanDavis]]
* 00:36 [[etherpad:p/U84-IdW1WfD2vZ4ogSCx]] was archived as [[etherpadbackup:U84-IdW1WfD2vZ4ogSCx]] by [[User:BryanDavis|BryanDavis]]
* 00:36 [[etherpad:p/U53g0zdCtb]] was archived as [[etherpadbackup:U53g0zdCtb]] by [[User:BryanDavis|BryanDavis]]
* 00:35 [[etherpad:p/U]] was archived as [[etherpadbackup:U]] by [[User:BryanDavis|BryanDavis]]
* 00:35 [[etherpad:p/Trusty_grid_shutdown]] was archived as [[etherpadbackup:Trusty_grid_shutdown]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/TransclusionModelIntroduction]] was archived as [[etherpadbackup:TransclusionModelIntroduction]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/Traffic-2021-08-24]] was archived as [[etherpadbackup:Traffic-2021-08-24]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/Traffic-2021-08-17]] was archived as [[etherpadbackup:Traffic-2021-08-17]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/Traffic-2021-02-17]] was archived as [[etherpadbackup:Traffic-2021-02-17]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/Traffic-2020-12-16]] was archived as [[etherpadbackup:Traffic-2020-12-16]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/Traffic-2020-11-02]] was archived as [[etherpadbackup:Traffic-2020-11-02]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/Traffic-2020-10-13]] was archived as [[etherpadbackup:Traffic-2020-10-13]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/Traffic-2020-09-28]] was archived as [[etherpadbackup:Traffic-2020-09-28]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/Traffic-2018-06-28]] was archived as [[etherpadbackup:Traffic-2018-06-28]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/Traffic-2018-06-14]] was archived as [[etherpadbackup:Traffic-2018-06-14]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/ToolLabs_Survey_feedback]] was archived as [[etherpadbackup:ToolLabs_Survey_feedback]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/TextInputWidgetValidation]] was archived as [[etherpadbackup:TextInputWidgetValidation]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/Templates]] was archived as [[etherpadbackup:Templates]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/Technical_Engagment_All_Hands_2019doing_and_should]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019doing_and_should]] by [[User:BryanDavis|BryanDavis]]
* 00:27 [[etherpad:p/Technical_Engagment_All_Hands_2019]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019]] by [[User:BryanDavis|BryanDavis]]
* 00:27 [[etherpad:p/TechOps-goals-Q4-FY15-16]] was archived as [[etherpadbackup:TechOps-goals-Q4-FY15-16]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/TechOps-goals-FQ3-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ3-FY1617]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/TechOps-goals-FQ2-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ2-FY1617]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/TechOps-goals-FQ1-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ1-FY1617]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/TechOps-2017-09-18]] was archived as [[etherpadbackup:TechOps-2017-09-18]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/TechOps-2015-10-14]] was archived as [[etherpadbackup:TechOps-2015-10-14]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/TechOps-2015-07-27]] was archived as [[etherpadbackup:TechOps-2015-07-27]] by [[User:BryanDavis|BryanDavis]]
* 00:23 [[etherpad:p/TechDaysCareers]] was archived as [[etherpadbackup:TechDaysCareers]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/TechDays-Mobile]] was archived as [[etherpadbackup:TechDays-Mobile]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/TargetLoader]] was archived as [[etherpadbackup:TargetLoader]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/T_movie]] was archived as [[etherpadbackup:T_movie]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/TFMQF]] was archived as [[etherpadbackup:TFMQF]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/T404584-announcement]] was archived as [[etherpadbackup:T404584-announcement]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/T395855]] was archived as [[etherpadbackup:T395855]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/T372912]] was archived as [[etherpadbackup:T372912]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/T364459]] was archived as [[etherpadbackup:T364459]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/T340241-new-fingerprints]] was archived as [[etherpadbackup:T340241-new-fingerprints]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/T340241-new-bastion]] was archived as [[etherpadbackup:T340241-new-bastion]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/T221119_-_log]] was archived as [[etherpadbackup:T221119_-_log]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/T191532]] was archived as [[etherpadbackup:T191532]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/T182722-outage]] was archived as [[etherpadbackup:T182722-outage]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/T152122_notes]] was archived as [[etherpadbackup:T152122_notes]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/T128190]] was archived as [[etherpadbackup:T128190]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/Swift-Switch-Originals]] was archived as [[etherpadbackup:Swift-Switch-Originals]] by [[User:BryanDavis|BryanDavis]]
* 00:14 [[etherpad:p/Summary/_Hypothesis]] was archived as [[etherpadbackup:Summary/_Hypothesis]] by [[User:BryanDavis|BryanDavis]]
* 00:14 [[etherpad:p/SquidToVarnish]] was archived as [[etherpadbackup:SquidToVarnish]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/Sqoop]] was archived as [[etherpadbackup:Sqoop]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/Sprint_D]] was archived as [[etherpadbackup:Sprint_D]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/ShutdownPreciseGrid]] was archived as [[etherpadbackup:ShutdownPreciseGrid]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/ServiceOps-AugustReboots]] was archived as [[etherpadbackup:ServiceOps-AugustReboots]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/ServiceOps]] was archived as [[etherpadbackup:ServiceOps]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/Searchfdbksurvey]] was archived as [[etherpadbackup:Searchfdbksurvey]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/Search_UserStories]] was archived as [[etherpadbackup:Search_UserStories]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/ScrummasterMeetup-2013-10-10]] was archived as [[etherpadbackup:ScrummasterMeetup-2013-10-10]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/Scrum-of-Scrums-copypaste]] was archived as [[etherpadbackup:Scrum-of-Scrums-copypaste]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/SandboxHatesMe]] was archived as [[etherpadbackup:SandboxHatesMe]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/SWAT_2016-04-14]] was archived as [[etherpadbackup:SWAT_2016-04-14]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/SWAT]] was archived as [[etherpadbackup:SWAT]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/SVImxzLIyw]] was archived as [[etherpadbackup:SVImxzLIyw]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/SRE-goals-FQ4-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ4-FY1819]] by [[User:BryanDavis|BryanDavis]]
* 00:06 [[etherpad:p/SRE-goals-FQ3-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ3-FY1819]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/SRE-goals-FQ1-FY1920]] was archived as [[etherpadbackup:SRE-goals-FQ1-FY1920]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/SRE-ServiceOps-StatusMeeting]] was archived as [[etherpadbackup:SRE-ServiceOps-StatusMeeting]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/SRE-ServiceOps-2019-01-03]] was archived as [[etherpadbackup:SRE-ServiceOps-2019-01-03]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/SRE-Foundations-2021-10-25]] was archived as [[etherpadbackup:SRE-Foundations-2021-10-25]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/SRE-Foundations-2020-07-01]] was archived as [[etherpadbackup:SRE-Foundations-2020-07-01]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/SRE-Foundations-2020-03-11]] was archived as [[etherpadbackup:SRE-Foundations-2020-03-11]] by [[User:BryanDavis|BryanDavis]]
* 00:02 [[etherpad:p/SRE-Foundations-2020-02-12]] was archived as [[etherpadbackup:SRE-Foundations-2020-02-12]] by [[User:BryanDavis|BryanDavis]]
* 00:02 [[etherpad:p/SRE-Foundations-2019-11-13]] was archived as [[etherpadbackup:SRE-Foundations-2019-11-13]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/SRE-Foundations-2019-08-07]] was archived as [[etherpadbackup:SRE-Foundations-2019-08-07]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/SRE-Foundations-2019-07-31]] was archived as [[etherpadbackup:SRE-Foundations-2019-07-31]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/SOAAuth]] was archived as [[etherpadbackup:SOAAuth]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/SGE_rebuild]] was archived as [[etherpadbackup:SGE_rebuild]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-05 ===
* 23:59 [[etherpad:p/S]] was archived as [[etherpadbackup:S]] by [[User:BryanDavis|BryanDavis]]
* 23:59 [[etherpad:p/RobotsMetaTag]] was archived as [[etherpadbackup:RobotsMetaTag]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/ResourceLoader-for-mobile]] was archived as [[etherpadbackup:ResourceLoader-for-mobile]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/ResourceLoader-Mobile]] was archived as [[etherpadbackup:ResourceLoader-Mobile]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/ResourceLoader]] was archived as [[etherpadbackup:ResourceLoader]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/ReleaseNotes]] was archived as [[etherpadbackup:ReleaseNotes]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/ReferenceInspector]] was archived as [[etherpadbackup:ReferenceInspector]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/Reading_web_sprint_2_names]] was archived as [[etherpadbackup:Reading_web_sprint_2_names]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/ReadingWebQ3Planning]] was archived as [[etherpadbackup:ReadingWebQ3Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/ReadingWebCodeReviewSessions]] was archived as [[etherpadbackup:ReadingWebCodeReviewSessions]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/ReadingShowcase]] was archived as [[etherpadbackup:ReadingShowcase]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/Reading]] was archived as [[etherpadbackup:Reading]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/RdvI3eLIj5AMM4PN_hHz]] was archived as [[etherpadbackup:RdvI3eLIj5AMM4PN_hHz]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/RWW-WikipediaZero]] was archived as [[etherpadbackup:RWW-WikipediaZero]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/RLRLRL]] was archived as [[etherpadbackup:RLRLRL]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/RLFuncDeps]] was archived as [[etherpadbackup:RLFuncDeps]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/RLExceptions]] was archived as [[etherpadbackup:RLExceptions]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/RL-mwloader-testcases-tmp]] was archived as [[etherpadbackup:RL-mwloader-testcases-tmp]] by [[User:BryanDavis|BryanDavis]]
* 23:50 [[etherpad:p/RL-mwloader-testcases]] was archived as [[etherpadbackup:RL-mwloader-testcases]] by [[User:BryanDavis|BryanDavis]]
* 23:50 [[etherpad:p/RL-async-head]] was archived as [[etherpadbackup:RL-async-head]] by [[User:BryanDavis|BryanDavis]]
* 23:49 [[etherpad:p/RD201412]] was archived as [[etherpadbackup:RD201412]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/RD201411]] was archived as [[etherpadbackup:RD201411]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/RD201409]] was archived as [[etherpadbackup:RD201409]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/RD2014]] was archived as [[etherpadbackup:RD2014]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/Quarterly_metrics_scorecard]] was archived as [[etherpadbackup:Quarterly_metrics_scorecard]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/QgiZvx5mWMQzlM03zCC1]] was archived as [[etherpadbackup:QgiZvx5mWMQzlM03zCC1]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/QR1]] was archived as [[etherpadbackup:QR1]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/QPP]] was archived as [[etherpadbackup:QPP]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/QA_Automation_sync_up]] was archived as [[etherpadbackup:QA_Automation_sync_up]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/Q]] was archived as [[etherpadbackup:Q]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/Puppet3]] was archived as [[etherpadbackup:Puppet3]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/ProofreadPageLua]] was archived as [[etherpadbackup:ProofreadPageLua]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/Phragile-requirements-2015-05]] was archived as [[etherpadbackup:Phragile-requirements-2015-05]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/Phabricator_Upgrade_Planning_20190123]] was archived as [[etherpadbackup:Phabricator_Upgrade_Planning_20190123]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/PhabricatorInstall]] was archived as [[etherpadbackup:PhabricatorInstall]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/PerformanceUpdate201601]] was archived as [[etherpadbackup:PerformanceUpdate201601]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/PerformanceDecemberUpdate]] was archived as [[etherpadbackup:PerformanceDecemberUpdate]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/Performance-2015-12-14]] was archived as [[etherpadbackup:Performance-2015-12-14]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/Performance]] was archived as [[etherpadbackup:Performance]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/PartnerEngineerTraining]] was archived as [[etherpadbackup:PartnerEngineerTraining]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/PV_Analytics_Infra]] was archived as [[etherpadbackup:PV_Analytics_Infra]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/PVTransition]] was archived as [[etherpadbackup:PVTransition]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/PAWS-2020-04-28]] was archived as [[etherpadbackup:PAWS-2020-04-28]] by [[User:BryanDavis|BryanDavis]]
* 23:37 [[etherpad:p/P&amp;EDashboard20160112]] was archived as [[etherpadbackup:P&amp;EDashboard20160112]] by [[User:BryanDavis|BryanDavis]]
* 23:37 [[etherpad:p/P]] was archived as [[etherpadbackup:P]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/OsJfzQH80T]] was archived as [[etherpadbackup:OsJfzQH80T]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/OpsSummit2013Day2]] was archived as [[etherpadbackup:OpsSummit2013Day2]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/OldWmfBranchCleanup]] was archived as [[etherpadbackup:OldWmfBranchCleanup]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/OctoberMetricsAgenda]] was archived as [[etherpadbackup:OctoberMetricsAgenda]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/OSM-Wikimedia]] was archived as [[etherpadbackup:OSM-Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/OOjsUI-TODO]] was archived as [[etherpadbackup:OOjsUI-TODO]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/NewMobileTech]] was archived as [[etherpadbackup:NewMobileTech]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/NewDeploysToday]] was archived as [[etherpadbackup:NewDeploysToday]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/N_sprint]] was archived as [[etherpadbackup:N_sprint]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/NQpvqJeiKg]] was archived as [[etherpadbackup:NQpvqJeiKg]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/N5q9EmRswClmLQyBOezv]] was archived as [[etherpadbackup:N5q9EmRswClmLQyBOezv]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/Mobile_at_Wikimania_2015]] was archived as [[etherpadbackup:Mobile_at_Wikimania_2015]] by [[User:BryanDavis|BryanDavis]]
* 23:30 [[etherpad:p/Mobile_Parsoid]] was archived as [[etherpadbackup:Mobile_Parsoid]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/MobileWeb_Iteration_name]] was archived as [[etherpadbackup:MobileWeb_Iteration_name]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/MobileWebOnboarding]] was archived as [[etherpadbackup:MobileWebOnboarding]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/MobileWebDevelopment]] was archived as [[etherpadbackup:MobileWebDevelopment]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/MobileWeb-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Retrospective]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/MobileWeb-Q4-Planning]] was archived as [[etherpadbackup:MobileWeb-Q4-Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/MobileWeb-Q3-Planning]] was archived as [[etherpadbackup:MobileWeb-Q3-Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:26 [[etherpad:p/MobileWeb-Q2-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Q2-Retrospective]] by [[User:BryanDavis|BryanDavis]]
* 23:26 [[etherpad:p/MobileWeb-IterationName]] was archived as [[etherpadbackup:MobileWeb-IterationName]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/MobileWeb-CodeReview-Discussion]] was archived as [[etherpadbackup:MobileWeb-CodeReview-Discussion]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/MobileWatchlist]] was archived as [[etherpadbackup:MobileWatchlist]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/MobileUX]] was archived as [[etherpadbackup:MobileUX]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/MobileTutorial]] was archived as [[etherpadbackup:MobileTutorial]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/MobileToCore]] was archived as [[etherpadbackup:MobileToCore]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/MobileTechDays]] was archived as [[etherpadbackup:MobileTechDays]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/MobileTeamLunchHangout]] was archived as [[etherpadbackup:MobileTeamLunchHangout]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/MobileTeamHangout]] was archived as [[etherpadbackup:MobileTeamHangout]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/MobileShowcase]] was archived as [[etherpadbackup:MobileShowcase]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/MobileResourcePlanning]] was archived as [[etherpadbackup:MobileResourcePlanning]] by [[User:BryanDavis|BryanDavis]]
* 23:20 [[etherpad:p/MobileQ1_2014_2015]] was archived as [[etherpadbackup:MobileQ1_2014_2015]] by [[User:BryanDavis|BryanDavis]]
* 23:20 [[etherpad:p/MobileProjectPlanning-Sept2012]] was archived as [[etherpadbackup:MobileProjectPlanning-Sept2012]] by [[User:BryanDavis|BryanDavis]]
* 23:19 [[etherpad:p/MobilePageLoadStats]] was archived as [[etherpadbackup:MobilePageLoadStats]] by [[User:BryanDavis|BryanDavis]]
* 23:19 [[etherpad:p/MobileOps]] was archived as [[etherpadbackup:MobileOps]] by [[User:BryanDavis|BryanDavis]]
* 23:18 [[etherpad:p/MobileOperatingSystems]] was archived as [[etherpadbackup:MobileOperatingSystems]] by [[User:BryanDavis|BryanDavis]]
* 23:18 [[etherpad:p/MobileFrontendCore]] was archived as [[etherpadbackup:MobileFrontendCore]] by [[User:BryanDavis|BryanDavis]]
* 23:17 [[etherpad:p/MobileFrontendChanges]] was archived as [[etherpadbackup:MobileFrontendChanges]] by [[User:BryanDavis|BryanDavis]]
* 23:17 [[etherpad:p/MobileFrontend-core]] was archived as [[etherpadbackup:MobileFrontend-core]] by [[User:BryanDavis|BryanDavis]]
* 23:16 [[etherpad:p/MobileFrontend-Deployment-Issues-20120423]] was archived as [[etherpadbackup:MobileFrontend-Deployment-Issues-20120423]] by [[User:BryanDavis|BryanDavis]]
* 23:16 [[etherpad:p/MobileFrontend-20120507-deployment-issues]] was archived as [[etherpadbackup:MobileFrontend-20120507-deployment-issues]] by [[User:BryanDavis|BryanDavis]]
* 23:15 [[etherpad:p/MobileDeploymentPainPoints]] was archived as [[etherpadbackup:MobileDeploymentPainPoints]] by [[User:BryanDavis|BryanDavis]]
* 23:15 [[etherpad:p/MobileChangeLog]] was archived as [[etherpadbackup:MobileChangeLog]] by [[User:BryanDavis|BryanDavis]]
* 23:14 [[etherpad:p/MobileAsks]] was archived as [[etherpadbackup:MobileAsks]] by [[User:BryanDavis|BryanDavis]]
* 23:13 [[etherpad:p/MobileArchitectureReview]] was archived as [[etherpadbackup:MobileArchitectureReview]] by [[User:BryanDavis|BryanDavis]]
* 23:13 [[etherpad:p/MobileAppTeam]] was archived as [[etherpadbackup:MobileAppTeam]] by [[User:BryanDavis|BryanDavis]]
* 23:12 [[etherpad:p/MobileAppRetrospective]] was archived as [[etherpadbackup:MobileAppRetrospective]] by [[User:BryanDavis|BryanDavis]]
* 23:12 [[etherpad:p/MobileAppReleaseManger]] was archived as [[etherpadbackup:MobileAppReleaseManger]] by [[User:BryanDavis|BryanDavis]]
* 23:11 [[etherpad:p/Mobile23Oct12]] was archived as [[etherpadbackup:Mobile23Oct12]] by [[User:BryanDavis|BryanDavis]]
* 23:11 [[etherpad:p/Mobile-WMDS]] was archived as [[etherpadbackup:Mobile-WMDS]] by [[User:BryanDavis|BryanDavis]]
* 23:10 [[etherpad:p/Mobile-VE-Integration-Planning]] was archived as [[etherpadbackup:Mobile-VE-Integration-Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:10 [[etherpad:p/Mobile-RL]] was archived as [[etherpadbackup:Mobile-RL]] by [[User:BryanDavis|BryanDavis]]
* 23:09 [[etherpad:p/Mobile-20navigation-20planning]] was archived as [[etherpadbackup:Mobile-20navigation-20planning]] by [[User:BryanDavis|BryanDavis]]
* 23:09 [[etherpad:p/Mm3uNugLtk]] was archived as [[etherpadbackup:Mm3uNugLtk]] by [[User:BryanDavis|BryanDavis]]
* 23:08 [[etherpad:p/Metrics_page]] was archived as [[etherpadbackup:Metrics_page]] by [[User:BryanDavis|BryanDavis]]
* 23:08 [[etherpad:p/Meetup]] was archived as [[etherpadbackup:Meetup]] by [[User:BryanDavis|BryanDavis]]
* 23:07 [[etherpad:p/MediaWiki_Workshop_Buea_-_Participants_emails]] was archived as [[etherpadbackup:MediaWiki_Workshop_Buea_-_Participants_emails]] by [[User:BryanDavis|BryanDavis]]
* 23:07 [[etherpad:p/MediaWiki_UI_Standardization_hack4]] was archived as [[etherpadbackup:MediaWiki_UI_Standardization_hack4]] by [[User:BryanDavis|BryanDavis]]
* 23:06 [[etherpad:p/McUX]] was archived as [[etherpadbackup:McUX]] by [[User:BryanDavis|BryanDavis]]
* 23:06 [[etherpad:p/Maps_pad]] was archived as [[etherpadbackup:Maps_pad]] by [[User:BryanDavis|BryanDavis]]
* 23:05 [[etherpad:p/MWDS2015_multi-device_world]] was archived as [[etherpadbackup:MWDS2015_multi-device_world]] by [[User:BryanDavis|BryanDavis]]
* 23:05 [[etherpad:p/MWDS2015_developer_documentation]] was archived as [[etherpadbackup:MWDS2015_developer_documentation]] by [[User:BryanDavis|BryanDavis]]
* 23:04 [[etherpad:p/MWCore2015Jan]] was archived as [[etherpadbackup:MWCore2015Jan]] by [[User:BryanDavis|BryanDavis]]
* 23:04 [[etherpad:p/MFCommitGuidelines]] was archived as [[etherpadbackup:MFCommitGuidelines]] by [[User:BryanDavis|BryanDavis]]
* 23:03 [[etherpad:p/MF-betalabs-planning]] was archived as [[etherpadbackup:MF-betalabs-planning]] by [[User:BryanDavis|BryanDavis]]
* 23:03 [[etherpad:p/MDS_2015_SOA_plenary]] was archived as [[etherpadbackup:MDS_2015_SOA_plenary]] by [[User:BryanDavis|BryanDavis]]
* 23:02 [[etherpad:p/MDS_2015_SOA_and_operations]] was archived as [[etherpadbackup:MDS_2015_SOA_and_operations]] by [[User:BryanDavis|BryanDavis]]
* 23:02 [[etherpad:p/Luke081515]] was archived as [[etherpadbackup:Luke081515]] by [[User:BryanDavis|BryanDavis]]
* 23:01 [[etherpad:p/Logout_Confirmation_DRC]] was archived as [[etherpadbackup:Logout_Confirmation_DRC]] by [[User:BryanDavis|BryanDavis]]
* 23:01 [[etherpad:p/LibrarizationRFC]] was archived as [[etherpadbackup:LibrarizationRFC]] by [[User:BryanDavis|BryanDavis]]
* 23:00 [[etherpad:p/Libera-channel-queue]] was archived as [[etherpadbackup:Libera-channel-queue]] by [[User:BryanDavis|BryanDavis]]
* 23:00 [[etherpad:p/Legends_of_Lavanya]] was archived as [[etherpadbackup:Legends_of_Lavanya]] by [[User:BryanDavis|BryanDavis]]
* 22:59 [[etherpad:p/LabsUpgradePlans]] was archived as [[etherpadbackup:LabsUpgradePlans]] by [[User:BryanDavis|BryanDavis]]
* 22:59 [[etherpad:p/LabsIrcConf]] was archived as [[etherpadbackup:LabsIrcConf]] by [[User:BryanDavis|BryanDavis]]
* 22:58 [[etherpad:p/LabsAPI]] was archived as [[etherpadbackup:LabsAPI]] by [[User:BryanDavis|BryanDavis]]
* 22:58 [[etherpad:p/Labs-Incident-Report-20015-03-31]] was archived as [[etherpadbackup:Labs-Incident-Report-20015-03-31]] by [[User:BryanDavis|BryanDavis]]
* 22:57 [[etherpad:p/Labs-2017-03-14]] was archived as [[etherpadbackup:Labs-2017-03-14]] by [[User:BryanDavis|BryanDavis]]
* 22:57 [[etherpad:p/Labs-2017-03-06]] was archived as [[etherpadbackup:Labs-2017-03-06]] by [[User:BryanDavis|BryanDavis]]
* 22:56 [[etherpad:p/Labs-2017-02-22]] was archived as [[etherpadbackup:Labs-2017-02-22]] by [[User:BryanDavis|BryanDavis]]
* 22:56 [[etherpad:p/Labs-2017-01-23]] was archived as [[etherpadbackup:Labs-2017-01-23]] by [[User:BryanDavis|BryanDavis]]
* 22:55 [[etherpad:p/Labs-2016-12-12]] was archived as [[etherpadbackup:Labs-2016-12-12]] by [[User:BryanDavis|BryanDavis]]
* 22:54 [[etherpad:p/Labs-2016-11-07]] was archived as [[etherpadbackup:Labs-2016-11-07]] by [[User:BryanDavis|BryanDavis]]
* 22:54 [[etherpad:p/Labs-2016-10-31]] was archived as [[etherpadbackup:Labs-2016-10-31]] by [[User:BryanDavis|BryanDavis]]
* 22:53 [[etherpad:p/Labs-2016-09-19]] was archived as [[etherpadbackup:Labs-2016-09-19]] by [[User:BryanDavis|BryanDavis]]
* 22:53 [[etherpad:p/Labs-2016-09-06]] was archived as [[etherpadbackup:Labs-2016-09-06]] by [[User:BryanDavis|BryanDavis]]
* 22:52 [[etherpad:p/Labs-2016-08-29]] was archived as [[etherpadbackup:Labs-2016-08-29]] by [[User:BryanDavis|BryanDavis]]
* 22:52 [[etherpad:p/Labs-2016-08-01]] was archived as [[etherpadbackup:Labs-2016-08-01]] by [[User:BryanDavis|BryanDavis]]
* 22:51 [[etherpad:p/Labs-2016-07-25]] was archived as [[etherpadbackup:Labs-2016-07-25]] by [[User:BryanDavis|BryanDavis]]
* 22:51 [[etherpad:p/Labs-2016-07-11]] was archived as [[etherpadbackup:Labs-2016-07-11]] by [[User:BryanDavis|BryanDavis]]
* 22:50 [[etherpad:p/Labs-2016-06-06]] was archived as [[etherpadbackup:Labs-2016-06-06]] by [[User:BryanDavis|BryanDavis]]
* 22:50 [[etherpad:p/Labs-2016-05-23]] was archived as [[etherpadbackup:Labs-2016-05-23]] by [[User:BryanDavis|BryanDavis]]
* 22:49 [[etherpad:p/Labs-2016-05-16]] was archived as [[etherpadbackup:Labs-2016-05-16]] by [[User:BryanDavis|BryanDavis]]
* 22:49 [[etherpad:p/Labs-2016-05-09]] was archived as [[etherpadbackup:Labs-2016-05-09]] by [[User:BryanDavis|BryanDavis]]
* 22:48 [[etherpad:p/Labs-2016-05-02]] was archived as [[etherpadbackup:Labs-2016-05-02]] by [[User:BryanDavis|BryanDavis]]
* 22:48 [[etherpad:p/Labs-2016-04-25]] was archived as [[etherpadbackup:Labs-2016-04-25]] by [[User:BryanDavis|BryanDavis]]
* 22:47 [[etherpad:p/Labs-20150602]] was archived as [[etherpadbackup:Labs-20150602]] by [[User:BryanDavis|BryanDavis]]
* 22:47 [[etherpad:p/L]] was archived as [[etherpadbackup:L]] by [[User:BryanDavis|BryanDavis]]
* 22:46 [[etherpad:p/Kay@31]] was archived as [[etherpadbackup:Kay@31]] by [[User:BryanDavis|BryanDavis]]
* 22:46 [[etherpad:p/Kafka-udp2log]] was archived as [[etherpadbackup:Kafka-udp2log]] by [[User:BryanDavis|BryanDavis]]
* 22:45 [[etherpad:p/K]] was archived as [[etherpadbackup:K]] by [[User:BryanDavis|BryanDavis]]
* 22:45 [[etherpad:p/J]] was archived as [[etherpadbackup:J]] by [[User:BryanDavis|BryanDavis]]
* 22:44 [[etherpad:p/Introduction_to_Phabricator_(WikiCon)]] was archived as [[etherpadbackup:Introduction_to_Phabricator_(WikiCon)]] by [[User:BryanDavis|BryanDavis]]
* 22:44 [[etherpad:p/Intro_a_las_tec_del_movimiento_Wikimedia]] was archived as [[etherpadbackup:Intro_a_las_tec_del_movimiento_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 22:43 [[etherpad:p/InlineScripting]] was archived as [[etherpadbackup:InlineScripting]] by [[User:BryanDavis|BryanDavis]]
* 22:43 [[etherpad:p/Incident-Replay-Sessions]] was archived as [[etherpadbackup:Incident-Replay-Sessions]] by [[User:BryanDavis|BryanDavis]]
* 22:42 [[etherpad:p/IlszcZZEEq]] was archived as [[etherpadbackup:IlszcZZEEq]] by [[User:BryanDavis|BryanDavis]]
* 22:42 [[etherpad:p/Icon_standardisation]] was archived as [[etherpadbackup:Icon_standardisation]] by [[User:BryanDavis|BryanDavis]]
* 22:41 [[etherpad:p/INGLW4jCQk]] was archived as [[etherpadbackup:INGLW4jCQk]] by [[User:BryanDavis|BryanDavis]]
* 22:41 [[etherpad:p/IBrokeWikipediaList]] was archived as [[etherpadbackup:IBrokeWikipediaList]] by [[User:BryanDavis|BryanDavis]]
* 22:40 [[etherpad:p/I]] was archived as [[etherpadbackup:I]] by [[User:BryanDavis|BryanDavis]]
* 22:39 [[etherpad:p/HuggleCon-2-planning]] was archived as [[etherpadbackup:HuggleCon-2-planning]] by [[User:BryanDavis|BryanDavis]]
* 22:39 [[etherpad:p/HuggleCon-1-participants-list]] was archived as [[etherpadbackup:HuggleCon-1-participants-list]] by [[User:BryanDavis|BryanDavis]]
* 22:38 [[etherpad:p/Huggle-dev-pad]] was archived as [[etherpadbackup:Huggle-dev-pad]] by [[User:BryanDavis|BryanDavis]]
* 22:38 [[etherpad:p/Huggle-IRC-conference]] was archived as [[etherpadbackup:Huggle-IRC-conference]] by [[User:BryanDavis|BryanDavis]]
* 22:37 [[etherpad:p/Huggle]] was archived as [[etherpadbackup:Huggle]] by [[User:BryanDavis|BryanDavis]]
* 22:37 [[etherpad:p/HowToReleaseAnApp]] was archived as [[etherpadbackup:HowToReleaseAnApp]] by [[User:BryanDavis|BryanDavis]]
* 22:36 [[etherpad:p/Hiera]] was archived as [[etherpadbackup:Hiera]] by [[User:BryanDavis|BryanDavis]]
* 22:36 [[etherpad:p/HadoopEtherpad]] was archived as [[etherpadbackup:HadoopEtherpad]] by [[User:BryanDavis|BryanDavis]]
* 22:35 [[etherpad:p/Hackathon_Showcase_&amp;_Coolest_Tool_Awards]] was archived as [[etherpadbackup:Hackathon_Showcase_&amp;_Coolest_Tool_Awards]] by [[User:BryanDavis|BryanDavis]]
* 22:35 [[etherpad:p/HKzL6x5e6z]] was archived as [[etherpadbackup:HKzL6x5e6z]] by [[User:BryanDavis|BryanDavis]]
* 22:34 [[etherpad:p/HJRCD1b81bHXm-ZS-l9w]] was archived as [[etherpadbackup:HJRCD1b81bHXm-ZS-l9w]] by [[User:BryanDavis|BryanDavis]]
* 22:34 [[etherpad:p/Gwods9UzuoEMR79NM8nl]] was archived as [[etherpadbackup:Gwods9UzuoEMR79NM8nl]] by [[User:BryanDavis|BryanDavis]]
* 22:33 [[etherpad:p/GroupSelectOption]] was archived as [[etherpadbackup:GroupSelectOption]] by [[User:BryanDavis|BryanDavis]]
* 22:33 [[etherpad:p/Google_Slack_Zoom_YouTube_bugs_in_Movement]] was archived as [[etherpadbackup:Google_Slack_Zoom_YouTube_bugs_in_Movement]] by [[User:BryanDavis|BryanDavis]]
* 22:32 [[etherpad:p/GitWorkflow]] was archived as [[etherpadbackup:GitWorkflow]] by [[User:BryanDavis|BryanDavis]]
* 22:32 [[etherpad:p/Git-migration-tools]] was archived as [[etherpadbackup:Git-migration-tools]] by [[User:BryanDavis|BryanDavis]]
* 22:31 [[etherpad:p/Git-migration]] was archived as [[etherpadbackup:Git-migration]] by [[User:BryanDavis|BryanDavis]]
* 22:31 [[etherpad:p/Git-Feb13]] was archived as [[etherpadbackup:Git-Feb13]] by [[User:BryanDavis|BryanDavis]]
* 22:30 [[etherpad:p/Gerrit_login_problems]] was archived as [[etherpadbackup:Gerrit_login_problems]] by [[User:BryanDavis|BryanDavis]]
* 22:30 [[etherpad:p/Gerrit_API_Wikimedia]] was archived as [[etherpadbackup:Gerrit_API_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 22:29 [[etherpad:p/GerritCleanupOrg]] was archived as [[etherpadbackup:GerritCleanupOrg]] by [[User:BryanDavis|BryanDavis]]
* 22:29 [[etherpad:p/GerritAccounts]] was archived as [[etherpadbackup:GerritAccounts]] by [[User:BryanDavis|BryanDavis]]
* 22:28 [[etherpad:p/GeoData]] was archived as [[etherpadbackup:GeoData]] by [[User:BryanDavis|BryanDavis]]
* 22:28 [[etherpad:p/Gather_Q4]] was archived as [[etherpadbackup:Gather_Q4]] by [[User:BryanDavis|BryanDavis]]
* 22:27 [[etherpad:p/G__________________]] was archived as [[etherpadbackup:G__________________]] by [[User:BryanDavis|BryanDavis]]
* 22:27 [[etherpad:p/GWToolsetReview]] was archived as [[etherpadbackup:GWToolsetReview]] by [[User:BryanDavis|BryanDavis]]
* 22:26 [[etherpad:p/G]] was archived as [[etherpadbackup:G]] by [[User:BryanDavis|BryanDavis]]
* 22:26 [[etherpad:p/FutureOfEditingTalk]] was archived as [[etherpadbackup:FutureOfEditingTalk]] by [[User:BryanDavis|BryanDavis]]
* 22:25 [[etherpad:p/FrontEndDevOnboarding-bahodir]] was archived as [[etherpadbackup:FrontEndDevOnboarding-bahodir]] by [[User:BryanDavis|BryanDavis]]
* 22:25 [[etherpad:p/FrontEndDevOnboarding]] was archived as [[etherpadbackup:FrontEndDevOnboarding]] by [[User:BryanDavis|BryanDavis]]
* 22:24 [[etherpad:p/France_-_communes_-_Data_.tab]] was archived as [[etherpadbackup:France_-_communes_-_Data_.tab]] by [[User:BryanDavis|BryanDavis]]
* 22:24 [[etherpad:p/Foo]] was archived as [[etherpadbackup:Foo]] by [[User:BryanDavis|BryanDavis]]
* 22:23 [[etherpad:p/FlowTodo]] was archived as [[etherpadbackup:FlowTodo]] by [[User:BryanDavis|BryanDavis]]
* 22:23 [[etherpad:p/FlowAsAService]] was archived as [[etherpadbackup:FlowAsAService]] by [[User:BryanDavis|BryanDavis]]
* 22:22 [[etherpad:p/Flow-ops-l_mail]] was archived as [[etherpadbackup:Flow-ops-l_mail]] by [[User:BryanDavis|BryanDavis]]
* 22:21 [[etherpad:p/First_URLs]] was archived as [[etherpadbackup:First_URLs]] by [[User:BryanDavis|BryanDavis]]
* 22:21 [[etherpad:p/Feb6Outage]] was archived as [[etherpadbackup:Feb6Outage]] by [[User:BryanDavis|BryanDavis]]
* 22:20 [[etherpad:p/Feb-6-Issues]] was archived as [[etherpadbackup:Feb-6-Issues]] by [[User:BryanDavis|BryanDavis]]
* 22:20 [[etherpad:p/FeaturesTeam2012-W15]] was archived as [[etherpadbackup:FeaturesTeam2012-W15]] by [[User:BryanDavis|BryanDavis]]
* 22:19 [[etherpad:p/FeaturesTeam2012-W14]] was archived as [[etherpadbackup:FeaturesTeam2012-W14]] by [[User:BryanDavis|BryanDavis]]
* 22:19 [[etherpad:p/FeaturesTeam2012-W13]] was archived as [[etherpadbackup:FeaturesTeam2012-W13]] by [[User:BryanDavis|BryanDavis]]
* 22:18 [[etherpad:p/FeaturesTeam2012-W08]] was archived as [[etherpadbackup:FeaturesTeam2012-W08]] by [[User:BryanDavis|BryanDavis]]
* 22:18 [[etherpad:p/FeaturesTeam2012-W07]] was archived as [[etherpadbackup:FeaturesTeam2012-W07]] by [[User:BryanDavis|BryanDavis]]
* 22:17 [[etherpad:p/FeaturesTeam2012-W06]] was archived as [[etherpadbackup:FeaturesTeam2012-W06]] by [[User:BryanDavis|BryanDavis]]
* 22:17 [[etherpad:p/FeaturesTeam2012-W03]] was archived as [[etherpadbackup:FeaturesTeam2012-W03]] by [[User:BryanDavis|BryanDavis]]
* 22:16 [[etherpad:p/FeaturesTeam2012-W011]] was archived as [[etherpadbackup:FeaturesTeam2012-W011]] by [[User:BryanDavis|BryanDavis]]
* 22:16 [[etherpad:p/FeaturesTeam20111227]] was archived as [[etherpadbackup:FeaturesTeam20111227]] by [[User:BryanDavis|BryanDavis]]
* 22:15 [[etherpad:p/FeaturesTeam20111213]] was archived as [[etherpadbackup:FeaturesTeam20111213]] by [[User:BryanDavis|BryanDavis]]
* 22:15 [[etherpad:p/FeaturesTeam20111206]] was archived as [[etherpadbackup:FeaturesTeam20111206]] by [[User:BryanDavis|BryanDavis]]
* 22:14 [[etherpad:p/FeaturesTeam-Preload]] was archived as [[etherpadbackup:FeaturesTeam-Preload]] by [[User:BryanDavis|BryanDavis]]
* 22:14 [[etherpad:p/Farm101]] was archived as [[etherpadbackup:Farm101]] by [[User:BryanDavis|BryanDavis]]
* 22:13 [[etherpad:p/FRAnalyticsQ2FY16]] was archived as [[etherpadbackup:FRAnalyticsQ2FY16]] by [[User:BryanDavis|BryanDavis]]
* 22:13 [[etherpad:p/FG3gMpB1qc]] was archived as [[etherpadbackup:FG3gMpB1qc]] by [[User:BryanDavis|BryanDavis]]
* 22:12 [[etherpad:p/F2pDHw25sm7eJS1Q3Tk2]] was archived as [[etherpadbackup:F2pDHw25sm7eJS1Q3Tk2]] by [[User:BryanDavis|BryanDavis]]
* 22:12 [[etherpad:p/F2gsk6WTo1]] was archived as [[etherpadbackup:F2gsk6WTo1]] by [[User:BryanDavis|BryanDavis]]
* 22:11 [[etherpad:p/F]] was archived as [[etherpadbackup:F]] by [[User:BryanDavis|BryanDavis]]
* 22:11 [[etherpad:p/ErAnycbwZA]] was archived as [[etherpadbackup:ErAnycbwZA]] by [[User:BryanDavis|BryanDavis]]
* 22:10 [[etherpad:p/EpicIndiaItinerary]] was archived as [[etherpadbackup:EpicIndiaItinerary]] by [[User:BryanDavis|BryanDavis]]
* 22:10 [[etherpad:p/Encuentro_técnico_hispanohablante_Wikimania_2022]] was archived as [[etherpadbackup:Encuentro_técnico_hispanohablante_Wikimania_2022]] by [[User:BryanDavis|BryanDavis]]
* 22:09 [[etherpad:p/EchoNotification]] was archived as [[etherpadbackup:EchoNotification]] by [[User:BryanDavis|BryanDavis]]
* 22:09 [[etherpad:p/EQIAD-rollout-sequence]] was archived as [[etherpadbackup:EQIAD-rollout-sequence]] by [[User:BryanDavis|BryanDavis]]
* 22:08 [[etherpad:p/EL_python]] was archived as [[etherpadbackup:EL_python]] by [[User:BryanDavis|BryanDavis]]
* 22:08 [[etherpad:p/E3Analytics]] was archived as [[etherpadbackup:E3Analytics]] by [[User:BryanDavis|BryanDavis]]
* 22:07 [[etherpad:p/E3-2013-03-07-deploy]] was archived as [[etherpadbackup:E3-2013-03-07-deploy]] by [[User:BryanDavis|BryanDavis]]
* 22:07 [[etherpad:p/DrinkingShoppingList]] was archived as [[etherpadbackup:DrinkingShoppingList]] by [[User:BryanDavis|BryanDavis]]
* 22:06 [[etherpad:p/DownWithTwentyPercent]] was archived as [[etherpadbackup:DownWithTwentyPercent]] by [[User:BryanDavis|BryanDavis]]
* 22:06 [[etherpad:p/Discussion__Redefine_Wikimedia_Hackathons]] was archived as [[etherpadbackup:Discussion__Redefine_Wikimedia_Hackathons]] by [[User:BryanDavis|BryanDavis]]
* 22:05 [[etherpad:p/DevelopingMobileApps]] was archived as [[etherpadbackup:DevelopingMobileApps]] by [[User:BryanDavis|BryanDavis]]
* 22:05 [[etherpad:p/DevSummitMain]] was archived as [[etherpadbackup:DevSummitMain]] by [[User:BryanDavis|BryanDavis]]
* 22:04 [[etherpad:p/DevSum2015_designresearch]] was archived as [[etherpadbackup:DevSum2015_designresearch]] by [[User:BryanDavis|BryanDavis]]
* 22:03 [[etherpad:p/DeploymentPrepReConfig]] was archived as [[etherpadbackup:DeploymentPrepReConfig]] by [[User:BryanDavis|BryanDavis]]
* 22:03 [[etherpad:p/DSEorCDH4]] was archived as [[etherpadbackup:DSEorCDH4]] by [[User:BryanDavis|BryanDavis]]
* 22:02 [[etherpad:p/DRworkshopParkingLot]] was archived as [[etherpadbackup:DRworkshopParkingLot]] by [[User:BryanDavis|BryanDavis]]
* 22:02 [[etherpad:p/DRMethodsMenuSite]] was archived as [[etherpadbackup:DRMethodsMenuSite]] by [[User:BryanDavis|BryanDavis]]
* 22:01 [[etherpad:p/ContinuousIntegration]] was archived as [[etherpadbackup:ContinuousIntegration]] by [[User:BryanDavis|BryanDavis]]
* 22:01 [[etherpad:p/CollectionsNameA]] was archived as [[etherpadbackup:CollectionsNameA]] by [[User:BryanDavis|BryanDavis]]
* 22:00 [[etherpad:p/Code-in_student_meeting]] was archived as [[etherpadbackup:Code-in_student_meeting]] by [[User:BryanDavis|BryanDavis]]
* 22:00 [[etherpad:p/CmMGbzxbZw]] was archived as [[etherpadbackup:CmMGbzxbZw]] by [[User:BryanDavis|BryanDavis]]
* 21:59 [[etherpad:p/Chromium-Upstream]] was archived as [[etherpadbackup:Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]]
* 21:59 [[etherpad:p/CentralNoticeDiscussion]] was archived as [[etherpadbackup:CentralNoticeDiscussion]] by [[User:BryanDavis|BryanDavis]]
* 21:58 [[etherpad:p/CentralNotice]] was archived as [[etherpadbackup:CentralNotice]] by [[User:BryanDavis|BryanDavis]]
* 21:58 [[etherpad:p/CampaignAssociatedMixins]] was archived as [[etherpadbackup:CampaignAssociatedMixins]] by [[User:BryanDavis|BryanDavis]]
* 21:57 [[etherpad:p/C_________]] was archived as [[etherpadbackup:C_________]] by [[User:BryanDavis|BryanDavis]]
* 21:57 [[etherpad:p/CZg-suME0Sx-nnCfKIEW]] was archived as [[etherpadbackup:CZg-suME0Sx-nnCfKIEW]] by [[User:BryanDavis|BryanDavis]]
* 21:56 [[etherpad:p/CSF_Katie]] was archived as [[etherpadbackup:CSF_Katie]] by [[User:BryanDavis|BryanDavis]]
* 21:56 [[etherpad:p/CEE-l10n-levels]] was archived as [[etherpadbackup:CEE-l10n-levels]] by [[User:BryanDavis|BryanDavis]]
* 21:55 [[etherpad:p/C-Lessons]] was archived as [[etherpadbackup:C-Lessons]] by [[User:BryanDavis|BryanDavis]]
* 21:55 [[etherpad:p/BotLicensing]] was archived as [[etherpadbackup:BotLicensing]] by [[User:BryanDavis|BryanDavis]]
* 21:54 [[etherpad:p/BetaClusterpriorityworksync]] was archived as [[etherpadbackup:BetaClusterpriorityworksync]] by [[User:BryanDavis|BryanDavis]]
* 21:54 [[etherpad:p/ArchitectureCommittee-2015-02-04]] was archived as [[etherpadbackup:ArchitectureCommittee-2015-02-04]] by [[User:BryanDavis|BryanDavis]]
* 21:53 [[etherpad:p/ArchDoc]] was archived as [[etherpadbackup:ArchDoc]] by [[User:BryanDavis|BryanDavis]]
* 21:53 [[etherpad:p/AppsReadership]] was archived as [[etherpadbackup:AppsReadership]] by [[User:BryanDavis|BryanDavis]]
* 21:52 [[etherpad:p/AppsQ120142015]] was archived as [[etherpadbackup:AppsQ120142015]] by [[User:BryanDavis|BryanDavis]]
* 21:52 [[etherpad:p/AppsPhabelloMigration]] was archived as [[etherpadbackup:AppsPhabelloMigration]] by [[User:BryanDavis|BryanDavis]]
* 21:51 [[etherpad:p/AppsNav]] was archived as [[etherpadbackup:AppsNav]] by [[User:BryanDavis|BryanDavis]]
* 21:51 [[etherpad:p/AppsContentService]] was archived as [[etherpadbackup:AppsContentService]] by [[User:BryanDavis|BryanDavis]]
* 21:50 [[etherpad:p/AppUserTesting]] was archived as [[etherpadbackup:AppUserTesting]] by [[User:BryanDavis|BryanDavis]]
* 21:50 [[etherpad:p/App-20design-20tweaks]] was archived as [[etherpadbackup:App-20design-20tweaks]] by [[User:BryanDavis|BryanDavis]]
* 21:49 [[etherpad:p/Android_front-end_dev_task]] was archived as [[etherpadbackup:Android_front-end_dev_task]] by [[User:BryanDavis|BryanDavis]]
* 21:48 [[etherpad:p/Analytics_all_staff]] was archived as [[etherpadbackup:Analytics_all_staff]] by [[User:BryanDavis|BryanDavis]]
* 21:48 [[etherpad:p/Analytics_Wikimania]] was archived as [[etherpadbackup:Analytics_Wikimania]] by [[User:BryanDavis|BryanDavis]]
* 21:47 [[etherpad:p/AnalyticsPlanningMeetingNotes]] was archived as [[etherpadbackup:AnalyticsPlanningMeetingNotes]] by [[User:BryanDavis|BryanDavis]]
* 21:47 [[etherpad:p/AnalyticsMilestonesAndPriorities]] was archived as [[etherpadbackup:AnalyticsMilestonesAndPriorities]] by [[User:BryanDavis|BryanDavis]]
* 21:46 [[etherpad:p/AnalyticsDevStaff]] was archived as [[etherpadbackup:AnalyticsDevStaff]] by [[User:BryanDavis|BryanDavis]]
* 21:46 [[etherpad:p/Analytics-Reseach]] was archived as [[etherpadbackup:Analytics-Reseach]] by [[User:BryanDavis|BryanDavis]]
* 21:45 [[etherpad:p/Analytics-Pixel-Service]] was archived as [[etherpadbackup:Analytics-Pixel-Service]] by [[User:BryanDavis|BryanDavis]]
* 21:45 [[etherpad:p/Alpha-beta_cleaning]] was archived as [[etherpadbackup:Alpha-beta_cleaning]] by [[User:BryanDavis|BryanDavis]]
* 21:44 [[etherpad:p/Adv_Browser_Tests_-_T86593]] was archived as [[etherpadbackup:Adv_Browser_Tests_-_T86593]] by [[User:BryanDavis|BryanDavis]]
* 21:44 [[etherpad:p/Adedolapo_-_survey_for_content_Translation_users]] was archived as [[etherpadbackup:Adedolapo_-_survey_for_content_Translation_users]] by [[User:BryanDavis|BryanDavis]]
* 21:43 [[etherpad:p/Accessibility_WMDS]] was archived as [[etherpadbackup:Accessibility_WMDS]] by [[User:BryanDavis|BryanDavis]]
* 21:43 [[etherpad:p/AWMD_IRC-Meeting-13]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-13]] by [[User:BryanDavis|BryanDavis]]
* 21:42 [[etherpad:p/AWMD_IRC-Meeting-12]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-12]] by [[User:BryanDavis|BryanDavis]]
* 21:42 [[etherpad:p/AWMD_IRC-Meeting-11]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-11]] by [[User:BryanDavis|BryanDavis]]
* 21:41 [[etherpad:p/AWMD_IRC-Meeting-10]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-10]] by [[User:BryanDavis|BryanDavis]]
* 21:41 [[etherpad:p/API-blog-post]] was archived as [[etherpadbackup:API-blog-post]] by [[User:BryanDavis|BryanDavis]]
* 21:40 [[etherpad:p/AFT5LaunchTests]] was archived as [[etherpadbackup:AFT5LaunchTests]] by [[User:BryanDavis|BryanDavis]]
* 21:40 [[etherpad:p/AFT5]] was archived as [[etherpadbackup:AFT5]] by [[User:BryanDavis|BryanDavis]]
* 21:39 [[etherpad:p/9RFl8QljDZ]] was archived as [[etherpadbackup:9RFl8QljDZ]] by [[User:BryanDavis|BryanDavis]]
* 21:39 [[etherpad:p/92]] was archived as [[etherpadbackup:92]] by [[User:BryanDavis|BryanDavis]]
* 21:38 [[etherpad:p/8r8ihQ7GpX]] was archived as [[etherpadbackup:8r8ihQ7GpX]] by [[User:BryanDavis|BryanDavis]]
* 21:38 [[etherpad:p/8D1ml5Q1Sb]] was archived as [[etherpadbackup:8D1ml5Q1Sb]] by [[User:BryanDavis|BryanDavis]]
* 21:37 [[etherpad:p/88952]] was archived as [[etherpadbackup:88952]] by [[User:BryanDavis|BryanDavis]]
* 21:37 [[etherpad:p/7hDQxehZZW]] was archived as [[etherpadbackup:7hDQxehZZW]] by [[User:BryanDavis|BryanDavis]]
* 21:36 [[etherpad:p/7YAmoRjGSAvfJrj9V9sD]] was archived as [[etherpadbackup:7YAmoRjGSAvfJrj9V9sD]] by [[User:BryanDavis|BryanDavis]]
* 21:36 [[etherpad:p/7FtE43Tuhu]] was archived as [[etherpadbackup:7FtE43Tuhu]] by [[User:BryanDavis|BryanDavis]]
* 21:35 [[etherpad:p/77SMYoPXek]] was archived as [[etherpadbackup:77SMYoPXek]] by [[User:BryanDavis|BryanDavis]]
* 21:35 [[etherpad:p/6gR8aSREkz]] was archived as [[etherpadbackup:6gR8aSREkz]] by [[User:BryanDavis|BryanDavis]]
* 21:34 [[etherpad:p/58040]] was archived as [[etherpadbackup:58040]] by [[User:BryanDavis|BryanDavis]]
* 21:34 [[etherpad:p/503-eqiad-cirrussearch]] was archived as [[etherpadbackup:503-eqiad-cirrussearch]] by [[User:BryanDavis|BryanDavis]]
* 21:33 [[etherpad:p/4pKxno7DAI]] was archived as [[etherpadbackup:4pKxno7DAI]] by [[User:BryanDavis|BryanDavis]]
* 21:33 [[etherpad:p/4J8fHuL0yB]] was archived as [[etherpadbackup:4J8fHuL0yB]] by [[User:BryanDavis|BryanDavis]]
* 21:32 [[etherpad:p/4.1_launch_post-mortem]] was archived as [[etherpadbackup:4.1_launch_post-mortem]] by [[User:BryanDavis|BryanDavis]]
* 21:32 [[etherpad:p/20percent]] was archived as [[etherpadbackup:20percent]] by [[User:BryanDavis|BryanDavis]]
* 21:31 [[etherpad:p/2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] was archived as [[etherpadbackup:2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] by [[User:BryanDavis|BryanDavis]]
* 21:30 [[etherpad:p/2025-10-workers-reboot]] was archived as [[etherpadbackup:2025-10-workers-reboot]] by [[User:BryanDavis|BryanDavis]]
* 21:30 [[etherpad:p/2025-04_toolforge_monthly_update]] was archived as [[etherpadbackup:2025-04_toolforge_monthly_update]] by [[User:BryanDavis|BryanDavis]]
* 21:29 [[etherpad:p/2024_tekton_upgrade]] was archived as [[etherpadbackup:2024_tekton_upgrade]] by [[User:BryanDavis|BryanDavis]]
* 21:29 [[etherpad:p/2021-switchdc-testing]] was archived as [[etherpadbackup:2021-switchdc-testing]] by [[User:BryanDavis|BryanDavis]]
* 21:28 [[etherpad:p/2021-switchdc-prep]] was archived as [[etherpadbackup:2021-switchdc-prep]] by [[User:BryanDavis|BryanDavis]]
* 21:28 [[etherpad:p/20205-06-components-api-reviews]] was archived as [[etherpadbackup:20205-06-components-api-reviews]] by [[User:BryanDavis|BryanDavis]]
* 21:27 [[etherpad:p/2019-09-25-cloudvirt-load-rebalance]] was archived as [[etherpadbackup:2019-09-25-cloudvirt-load-rebalance]] by [[User:BryanDavis|BryanDavis]]
* 21:27 [[etherpad:p/2019-07-23-eqiad-asw-a]] was archived as [[etherpadbackup:2019-07-23-eqiad-asw-a]] by [[User:BryanDavis|BryanDavis]]
* 21:26 [[etherpad:p/2019-07-16-docker-registry]] was archived as [[etherpadbackup:2019-07-16-docker-registry]] by [[User:BryanDavis|BryanDavis]]
* 21:26 [[etherpad:p/20180829-labstore-reboots]] was archived as [[etherpadbackup:20180829-labstore-reboots]] by [[User:BryanDavis|BryanDavis]]
* 21:25 [[etherpad:p/20150519-Mailman]] was archived as [[etherpadbackup:20150519-Mailman]] by [[User:BryanDavis|BryanDavis]]
* 21:25 [[etherpad:p/2014-antoine-june-checkin]] was archived as [[etherpadbackup:2014-antoine-june-checkin]] by [[User:BryanDavis|BryanDavis]]
* 21:24 [[etherpad:p/2013-2014-Q2-MobileWeb]] was archived as [[etherpadbackup:2013-2014-Q2-MobileWeb]] by [[User:BryanDavis|BryanDavis]]
* 21:24 [[etherpad:p/2012-2013-Q1-Mobile-Dept]] was archived as [[etherpadbackup:2012-2013-Q1-Mobile-Dept]] by [[User:BryanDavis|BryanDavis]]
* 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]]
* 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]]
* 20:22 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-03 ===
* 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 13:56 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/WIKISOO3]] was archived as [[etherpadbackup:WIKISOO3]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/VertalingWikiIt]] was archived as [[etherpadbackup:VertalingWikiIt]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/BugTriage-Collection]] was archived as [[etherpadbackup:BugTriage-Collection]] by [[User:BryanDavis|BryanDavis]]
* 09:25 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 09:18 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 07:46 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 07:28 [[etherpad:p/prodlikevagrant-hackathon-planning]] was archived as [[etherpadbackup:prodlikevagrant-hackathon-planning]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-05-02 ===
* 13:01 [[etherpad:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] was archived as [[etherpadbackup:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 07:18 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 06:58 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-05-01 ===
* 18:12 [[etherpad:p/10myths]] was archived as [[etherpadbackup:p/10myths]] by [[User:Krinkle|Krinkle]]
* 15:30 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 14:50 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:Jon Harald Søby|Jon Harald Søby]]
* 14:42 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 13:51 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 13:16 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 13:07 [[etherpad:p/copyvios-todo]] was archived as [[etherpadbackup:p/copyvios-todo]] by [[User:Chlod|Chlod]]
* 10:05 [[etherpad:p/hhvmdeploy]] was archived as [[etherpadbackup:p/hhvmdeploy]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-04-29 ===
* 00:49 [[etherpad:p/better-security-patch-management]] was archived as [[etherpadbackup:p/better-security-patch-management]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 00:49 [[etherpad:p/rooks-questions-to-alex]] was archived as [[etherpadbackup:p/rooks-questions-to-alex]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-04-27 ===
* 04:44 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 03:46 [[etherpad:p/wmh2023-Overview_of_technical_areas_&amp;_projects]] was archived as [[etherpadbackup:p/wmh2023-Overview_of_technical_areas_&amp;_projects]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/open_meeting_COVID-19_18/04]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_18/04]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/open_meeting_COVID-19_04/05]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_04/05]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/aswikimeet/july-2024]] was archived as [[etherpadbackup:p/aswikimeet/july-2024]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/Wikisource_Meetup_29/09/2020]] was archived as [[etherpadbackup:p/Wikisource_Meetup_29/09/2020]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] was archived as [[etherpadbackup:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/Wikimedia_LGBT+/Governance/2023-10-13]] was archived as [[etherpadbackup:p/Wikimedia_LGBT+/Governance/2023-10-13]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopD]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopD]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopC]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopC]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopB]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopB]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopA]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopA]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Stolpersteine_nach_WikiData_(01/2025]] was archived as [[etherpadbackup:p/Stolpersteine_nach_WikiData_(01/2025]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Sommer_24/8,_1._August]] was archived as [[etherpadbackup:p/Sommer_24/8,_1._August]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] was archived as [[etherpadbackup:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Meetup/Cape_Town/Cape_Town_22]] was archived as [[etherpadbackup:p/Meetup/Cape_Town/Cape_Town_22]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] was archived as [[etherpadbackup:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] was archived as [[etherpadbackup:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] was archived as [[etherpadbackup:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_C]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_C]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_B]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_B]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_A]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_A]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] was archived as [[etherpadbackup:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] was archived as [[etherpadbackup:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Wikispore]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Wikispore]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Translatable_modules]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Translatable_modules]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Movement_Strategy]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Movement_Strategy]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Lua_templates_for_beginners]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Lua_templates_for_beginners]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Help_your_community_grow]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Help_your_community_grow]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/101_Ways_to_Contribute]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/101_Ways_to_Contribute]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/AsWikiMentoringProject2024/Oct-Dec]] was archived as [[etherpadbackup:p/AsWikiMentoringProject2024/Oct-Dec]] by [[User:BryanDavis|BryanDavis]]
* 03:16 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]]
* 03:15 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:wmh2023-Overview_of_technical_areas_%26_projects|wmh2023-Overview_of_technical_areas_%26_projects]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_20%2F04|open_meeting_COVID-19_20%2F04]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_18%2F04|open_meeting_COVID-19_18%2F04]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_04%2F05|open_meeting_COVID-19_04%2F05]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:aswikimeet%2Fjuly-2024|aswikimeet%2Fjuly-2024]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikisource_Meetup_29%2F09%2F2020|Wikisource_Meetup_29%2F09%2F2020]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020|Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedia_LGBT+%2FGovernance%2F2023-10-13|Wikimedia_LGBT+%2FGovernance%2F2023-10-13]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopD|WikiCendekia2026%2FWarkopD]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopC|WikiCendekia2026%2FWarkopC]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopB|WikiCendekia2026%2FWarkopB]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopA|WikiCendekia2026%2FWarkopA]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Summary%2F_Hypothesis|Summary%2F_Hypothesis]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(12%2F2024)|Stolpersteine_nach_WikiData_(12%2F2024)]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(01%2F2025|Stolpersteine_nach_WikiData_(01%2F2025]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Sommer_24%2F8,_1._August|Sommer_24%2F8,_1._August]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala|SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Meetup%2FCape_Town%2FCape_Town_22|Meetup%2FCape_Town%2FCape_Town_22]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk|Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23|Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Igbo_Wikimedians_Community%2FMovement_Charter_Commun|Igbo_Wikimedians_Community%2FMovement_Charter_Commun]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_C|GLAM-Treffen_2020%2FGruppe_C]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_B|GLAM-Treffen_2020%2FGruppe_B]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_A|GLAM-Treffen_2020%2FGruppe_A]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Commons_Photographers_30%2F05%2F2020|Commons_Photographers_30%2F05%2F2020]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:ChristianSW%2FNotizen|ChristianSW%2FNotizen]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202|CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FOutlook_CEE_Meeting_2022_V2|CEE_Talks%2FOutlook_CEE_Meeting_2022_V2]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FWikispore|CEE_Meeting_2020%2FWikispore]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FTranslatable_modules|CEE_Meeting_2020%2FTranslatable_modules]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref|CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FMovement_Strategy|CEE_Meeting_2020%2FMovement_Strategy]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FLua_templates_for_beginners|CEE_Meeting_2020%2FLua_templates_for_beginners]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FHelp_your_community_grow|CEE_Meeting_2020%2FHelp_your_community_grow]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2F101_Ways_to_Contribute|CEE_Meeting_2020%2F101_Ways_to_Contribute]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:AsWikiMentoringProject2024%2FOct-Dec|AsWikiMentoringProject2024%2FOct-Dec]]
* 01:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:51 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:49 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:43 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:05 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]

=== 2026-04-26 ===
* 23:25 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 21:58 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 17:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 17:22 [[etherpad:p/bd808 test/path/?query=param]] was archived as [[etherpadbackup:p/bd808 test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 17:22 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 16:17 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]

=== 2026-04-20 ===
* 02:42 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 02:36 [[etherpad:p/bd808-etherpad-backup-test]] was archived by [[User:BryanDavis]]</text>
      <sha1>e9n1vteqode24yc46ntog878u1dwo6j</sha1>
    </revision>
    <revision>
      <id>2435507</id>
      <parentid>2435493</parentid>
      <timestamp>2026-07-11T21:27:59Z</timestamp>
      <contributor>
        <username>EtherpadBackupBot</username>
        <id>54504</id>
      </contributor>
      <comment>Future_of_Wikimedia_2026-07-07 archived by Pppery</comment>
      <origin>2435507</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="193384" sha1="53a0u2zul9j4y6cxr6ynes3igynur60" xml:space="preserve">=== 2026-07-11 ===
* 21:27 [[etherpad:p/Future_of_Wikimedia_2026-07-07]] was archived as [[etherpadbackup:Future_of_Wikimedia_2026-07-07]] by [[User:Pppery|Pppery]]
* 14:02 [[etherpad:p/WikimediaEvents]] was archived as [[etherpadbackup:WikimediaEvents]] by [[User:Exec8|Exec8]]

=== 2026-07-06 ===
* 21:30 [[etherpad:p/MWStake]] was archived as [[etherpadbackup:MWStake]] by [[User:Pppery|Pppery]]

=== 2026-06-21 ===
* 15:11 [[etherpad:p/IN-KL_Meetup_April_2026]] was archived as [[etherpadbackup:IN-KL_Meetup_April_2026]] by [[User:Gnoeee|Gnoeee]]
* 15:11 [[etherpad:p/IN-KL_Meetup_May_2026]] was archived as [[etherpadbackup:IN-KL_Meetup_May_2026]] by [[User:Gnoeee|Gnoeee]]

=== 2026-06-15 ===
* 09:55 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Ericliu1912|Ericliu1912]]

=== 2026-06-13 ===
* 12:08 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Chlod|Chlod]]

=== 2026-06-08 ===
* 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]]
* 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]]

=== 2026-05-29 ===
* 03:47 [[etherpad:p/ESEAPCon26_sessions_csv]] was archived as [[etherpadbackup:ESEAPCon26_sessions_csv]] by [[User:Robertsky|Robertsky]]
* 03:41 [[etherpad:p/ESEAPCon26_day2_Room2_DB9UHD]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_DB9UHD]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_JD3H99]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_JD3H99]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_DBKL7W]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_DBKL7W]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_3KZYPS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_3KZYPS]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_YF7VEZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_YF7VEZ]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_8TXHAF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_8TXHAF]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_QRT8BV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_QRT8BV]] by [[User:Robertsky|Robertsky]]
* 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_RFAU8M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_RFAU8M]] by [[User:Robertsky|Robertsky]]
* 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_VU3QL8]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_VU3QL8]] by [[User:Robertsky|Robertsky]]
* 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_M3QUZD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_M3QUZD]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_TFFBBH]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_TFFBBH]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_PQYYHB]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_PQYYHB]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_CA7G3X]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_CA7G3X]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_8RX8NZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_8RX8NZ]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_JYNFKE]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_JYNFKE]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_LFV39K]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_LFV39K]] by [[User:Robertsky|Robertsky]]
* 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_H9JCWN]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_H9JCWN]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_9EF8DZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9EF8DZ]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_8EQWQL]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_8EQWQL]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_A87WBV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_A87WBV]] by [[User:Robertsky|Robertsky]]
* 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_3K3T9M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_3K3T9M]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_9DJJP7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9DJJP7]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_AKYUKC]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_AKYUKC]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_S99FWD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_S99FWD]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_HGHY9E]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_HGHY9E]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_YL8ST7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_YL8ST7]] by [[User:Robertsky|Robertsky]]
* 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_8ZNVWS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_8ZNVWS]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_C7ZRET]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_C7ZRET]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_NCHDPG]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_NCHDPG]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_TW9WFJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_TW9WFJ]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_VNX3UJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_VNX3UJ]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_BQHSPF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_BQHSPF]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_ZNUEAR]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_ZNUEAR]] by [[User:Robertsky|Robertsky]]
* 03:30 [[etherpad:p/ESEAPCon26_day2_Outofvenue_VVR7KR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_VVR7KR]] by [[User:Robertsky|Robertsky]]
* 03:29 [[etherpad:p/ESEAPCon26_day2_Outofvenue_CJTRQM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_CJTRQM]] by [[User:Robertsky|Robertsky]]
* 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_DEXKVE]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DEXKVE]] by [[User:Robertsky|Robertsky]]
* 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_7DSNAX]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7DSNAX]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_DZGGRB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DZGGRB]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_7MBXLG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7MBXLG]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_Q3XQYZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_Q3XQYZ]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_TDXWE3]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_TDXWE3]] by [[User:Robertsky|Robertsky]]
* 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_HZZZKL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_HZZZKL]] by [[User:Robertsky|Robertsky]]
* 03:27 [[etherpad:p/ESEAPCon26_day2_Room2_JTBXUG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JTBXUG]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_ALULGS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_ALULGS]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_MNBFWL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_MNBFWL]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_RRHPKR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_RRHPKR]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_AURUVZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_AURUVZ]] by [[User:Robertsky|Robertsky]]
* 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_3MAKHH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_3MAKHH]] by [[User:Robertsky|Robertsky]]
* 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_TXFRZB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_TXFRZB]] by [[User:Robertsky|Robertsky]]
* 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_QUFR97]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QUFR97]] by [[User:Robertsky|Robertsky]]
* 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_JT7TFH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JT7TFH]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_LJ3BL8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_LJ3BL8]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_9YDQCT]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_9YDQCT]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_GKN9F7]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_GKN9F7]] by [[User:Robertsky|Robertsky]]
* 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_CFUMZ9]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CFUMZ9]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_CRTN93]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CRTN93]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_MGZMP8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_MGZMP8]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_8WQJKB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_8WQJKB]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LLPZSB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LLPZSB]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_HMSJPM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_HMSJPM]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_SWCDHM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_SWCDHM]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_3CDL7X]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_3CDL7X]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_AHCNHB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_AHCNHB]] by [[User:Robertsky|Robertsky]]
* 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LTSVEH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LTSVEH]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_ZVYUWS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_ZVYUWS]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_T8YSUB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_T8YSUB]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_LMXU7L]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LMXU7L]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_YKMM9C]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_YKMM9C]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_RZKSWJ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_RZKSWJ]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Outofvenue_3RMDEL]] was archived as [[etherpadbackup:ESEAPCon26_day1_Outofvenue_3RMDEL]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room3_XAJUWY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room3_XAJUWY]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_FSZJK7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FSZJK7]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_W3ND3N]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_W3ND3N]] by [[User:Robertsky|Robertsky]]
* 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_8AR9U9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_8AR9U9]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_NUWQDM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_NUWQDM]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_DGZT88]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_DGZT88]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_FPWESV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FPWESV]] by [[User:Robertsky|Robertsky]]
* 03:15 [[etherpad:p/ESEAPCon26_day1_Room1_7EKDB9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_7EKDB9]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_XVFLQU]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_XVFLQU]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_KCR737]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_KCR737]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_EBATCZ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_EBATCZ]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_MGKRNF]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_MGKRNF]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_DHFDW7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_DHFDW7]] by [[User:Robertsky|Robertsky]]
* 03:14 [[etherpad:p/ESEAPCon26_day1_Mainroom_CLMDUY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_CLMDUY]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_B7Y3PM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_B7Y3PM]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_WGT9AQ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_WGT9AQ]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_AXG3KB]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_AXG3KB]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_L8GRH9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_L8GRH9]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_UAF8RA]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_UAF8RA]] by [[User:Robertsky|Robertsky]]
* 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_NTXNZV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_NTXNZV]] by [[User:Robertsky|Robertsky]]

=== 2026-05-26 ===
* 21:47 [[etherpad:p/Wikimedia_Hackathon_2026_Closing_Showcase]] was archived as [[etherpadbackup:Wikimedia_Hackathon_2026_Closing_Showcase]] by [[User:Gnoeee|Gnoeee]]
* 21:47 [[etherpad:p/South_Asia_Community_Call]] was archived as [[etherpadbackup:South_Asia_Community_Call]] by [[User:Gnoeee|Gnoeee]]

=== 2026-05-23 ===
* 09:18 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Ericliu1912|Ericliu1912]]
* 09:18 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Ericliu1912|Ericliu1912]]
* 02:46 [[etherpad:p/org_structure_reading_group]] was archived as [[etherpadbackup:org_structure_reading_group]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-18 ===
* 19:20 [[etherpad:p/Chartres]] was archived as [[etherpadbackup:Chartres]] by [[User:Sj|Sj]]
* 19:08 [[etherpad:p/city-homelessness-okn]] was archived as [[etherpadbackup:city-homelessness-okn]] by [[User:Sj|Sj]]
* 19:07 [[etherpad:p/WM-structural-problems-overview]] was archived as [[etherpadbackup:WM-structural-problems-overview]] by [[User:Sj|Sj]]
* 19:03 [[etherpad:p/WLM-2021-May]] was archived as [[etherpadbackup:WLM-2021-May]] by [[User:Sj|Sj]]
* 19:02 [[etherpad:p/WLM-2021-April]] was archived as [[etherpadbackup:WLM-2021-April]] by [[User:Sj|Sj]]
* 19:02 [[etherpad:p/shexcg_minutes]] was archived as [[etherpadbackup:shexcg_minutes]] by [[User:Sj|Sj]]
* 19:01 [[etherpad:p/ctp-letter]] was archived as [[etherpadbackup:ctp-letter]] by [[User:Sj|Sj]]
* 18:59 [[etherpad:p/Wikimania_oral_history]] was archived as [[etherpadbackup:Wikimania_oral_history]] by [[User:Sj|Sj]]
* 18:55 [[etherpad:p/covid19]] was archived as [[etherpadbackup:covid19]] by [[User:Sj|Sj]]
* 18:54 [[etherpad:p/Covering_George_Floyd]] was archived as [[etherpadbackup:Covering_George_Floyd]] by [[User:Sj|Sj]]
* 17:37 [[etherpad:p/JROSTS18-HACKDAY]] was archived as [[etherpadbackup:JROSTS18-HACKDAY]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFF]] was archived as [[etherpadbackup:JROSTS18-UNCFF]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFE]] was archived as [[etherpadbackup:JROSTS18-UNCFE]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFD]] was archived as [[etherpadbackup:JROSTS18-UNCFD]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFC]] was archived as [[etherpadbackup:JROSTS18-UNCFC]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCFB]] was archived as [[etherpadbackup:JROSTS18-UNCFB]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-OA.2]] was archived as [[etherpadbackup:JROSTS18-OA.2]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-UNCONFERENCING]] was archived as [[etherpadbackup:JROSTS18-UNCONFERENCING]] by [[User:Sj|Sj]]
* 17:36 [[etherpad:p/JROSTS18-JROST]] was archived as [[etherpadbackup:JROSTS18-JROST]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-OB.1]] was archived as [[etherpadbackup:JROSTS18-OB.1]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-EOA]] was archived as [[etherpadbackup:JROSTS18-EOA]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-WFC.3]] was archived as [[etherpadbackup:JROSTS18-WFC.3]] by [[User:Sj|Sj]]
* 17:35 [[etherpad:p/JROSTS18-WFC.2]] was archived as [[etherpadbackup:JROSTS18-WFC.2]] by [[User:Sj|Sj]]
* 17:34 [[etherpad:p/JROSTS18-WFA.2]] was archived as [[etherpadbackup:JROSTS18-WFA.2]] by [[User:Sj|Sj]]
* 17:34 [[etherpad:p/JROSTS18-WFA.1]] was archived as [[etherpadbackup:JROSTS18-WFA.1]] by [[User:Sj|Sj]]
* 17:33 [[etherpad:p/JROSTS18-CROSSCUTTING]] was archived as [[etherpadbackup:JROSTS18-CROSSCUTTING]] by [[User:Sj|Sj]]
* 17:33 [[etherpad:p/JROST-Master]] was archived as [[etherpadbackup:JROST-Master]] by [[User:Sj|Sj]]
* 17:33 [[etherpad:p/JROSTS18-WFC.1]] was archived as [[etherpadbackup:JROSTS18-WFC.1]] by [[User:Sj|Sj]]
* 17:19 [[etherpad:p/15.320]] was archived as [[etherpadbackup:15.320]] by [[User:Sj|Sj]]
* 17:17 [[etherpad:p/medical_hotspots]] was archived as [[etherpadbackup:medical_hotspots]] by [[User:Sj|Sj]]
* 17:16 [[etherpad:p/community_health]] was archived as [[etherpadbackup:community_health]] by [[User:Sj|Sj]]
* 17:14 [[etherpad:p/whatsnext]] was archived as [[etherpadbackup:whatsnext]] by [[User:Sj|Sj]]
* 17:12 [[etherpad:p/MWF20160318]] was archived as [[etherpadbackup:MWF20160318]] by [[User:Sj|Sj]]
* 17:12 [[etherpad:p/MWF20160226]] was archived as [[etherpadbackup:MWF20160226]] by [[User:Sj|Sj]]
* 17:11 [[etherpad:p/ceem14-pr]] was archived as [[etherpadbackup:ceem14-pr]] by [[User:Sj|Sj]]
* 16:54 [[etherpad:p/Templates,_Page_Components_and_editing]] was archived as [[etherpadbackup:Templates,_Page_Components_and_editing]] by [[User:Sj|Sj]]
* 16:54 [[etherpad:p/GLAM-Wikipedia-fr]] was archived as [[etherpadbackup:GLAM-Wikipedia-fr]] by [[User:Sj|Sj]]
* 16:49 [[etherpad:p/editcontentmodel]] was archived as [[etherpadbackup:editcontentmodel]] by [[User:Sj|Sj]]
* 16:43 [[etherpad:p/multimedia-weekly-meeting-2014-05-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-05-14]] by [[User:Sj|Sj]]
* 16:06 [[etherpad:p/cuny-wikiwed-2026-04]] was archived as [[etherpadbackup:cuny-wikiwed-2026-04]] by [[User:Pharos|Pharos]]
* 16:06 [[etherpad:p/cuny-wikiwed-2026-03]] was archived as [[etherpadbackup:cuny-wikiwed-2026-03]] by [[User:Pharos|Pharos]]
* 16:06 [[etherpad:p/cuny-wikiwed-2026-01]] was archived as [[etherpadbackup:cuny-wikiwed-2026-01]] by [[User:Pharos|Pharos]]
* 16:06 [[etherpad:p/cuny-wikiwed-2025-11]] was archived as [[etherpadbackup:cuny-wikiwed-2025-11]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-10]] was archived as [[etherpadbackup:cuny-wikiwed-2025-10]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-09]] was archived as [[etherpadbackup:cuny-wikiwed-2025-09]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-08]] was archived as [[etherpadbackup:cuny-wikiwed-2025-08]] by [[User:Pharos|Pharos]]
* 16:05 [[etherpad:p/cuny-wikiwed-2025-07]] was archived as [[etherpadbackup:cuny-wikiwed-2025-07]] by [[User:Pharos|Pharos]]
* 16:04 [[etherpad:p/cuny-wikiwed-2025-06]] was archived as [[etherpadbackup:cuny-wikiwed-2025-06]] by [[User:Pharos|Pharos]]
* 16:04 [[etherpad:p/cuny-wikiwed-2025-05]] was archived as [[etherpadbackup:cuny-wikiwed-2025-05]] by [[User:Pharos|Pharos]]
* 16:04 [[etherpad:p/cuny-wikiwed-2025-04]] was archived as [[etherpadbackup:cuny-wikiwed-2025-04]] by [[User:Pharos|Pharos]]

=== 2026-05-10 ===
* 18:10 [[etherpad:p/wmhackshowcase2020]] was archived as [[etherpadbackup:wmhackshowcase2020]] by [[User:BryanDavis|BryanDavis]]
* 18:03 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 18:01 [[etherpad:p/wdqs-500s]] was archived as [[etherpadbackup:wdqs-500s]] by [[User:BryanDavis|BryanDavis]]
* 18:00 [[etherpad:p/wep_data_task_force_meeting_#1]] was archived as [[etherpadbackup:wep_data_task_force_meeting_#1]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-07 ===
* 00:35 [[etherpad:p/zh-ve-screenshots]] was archived as [[etherpadbackup:zh-ve-screenshots]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/zerocrash]] was archived as [[etherpadbackup:zerocrash]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/zero-mf-sync-up]] was archived as [[etherpadbackup:zero-mf-sync-up]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/yuvidump]] was archived as [[etherpadbackup:yuvidump]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/yuvi-knowledge-transfer]] was archived as [[etherpadbackup:yuvi-knowledge-transfer]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/yeZfmYkMu2]] was archived as [[etherpadbackup:yeZfmYkMu2]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/yAqOuwkDKuhwhwjvd44j]] was archived as [[etherpadbackup:yAqOuwkDKuhwhwjvd44j]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/y4gqz6&lt;fe46Q]] was archived as [[etherpadbackup:y4gqz6&lt;fe46Q]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/y4gqz6&lt;fe46Q]] was archived as [[etherpadbackup:y4gqz6&lt;fe46Q]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/xZLrPEASWV]] was archived as [[etherpadbackup:xZLrPEASWV]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/x-analytics]] was archived as [[etherpadbackup:x-analytics]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/wtf-is-production]] was archived as [[etherpadbackup:wtf-is-production]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/wtf]] was archived as [[etherpadbackup:wtf]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/wp-app-1-1-release-post]] was archived as [[etherpadbackup:wp-app-1-1-release-post]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/wmhack2021-phab]] was archived as [[etherpadbackup:wmhack2021-phab]] by [[User:BryanDavis|BryanDavis]]
* 00:27 [[etherpad:p/wmhack2017-ciw]] was archived as [[etherpadbackup:wmhack2017-ciw]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/wmhack15-wikidata-architecture-session]] was archived as [[etherpadbackup:wmhack15-wikidata-architecture-session]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/wmh2024-leaving-coordination]] was archived as [[etherpadbackup:wmh2024-leaving-coordination]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/wmh2023-Toolhub,_Toolhunt,_and_the_quest]] was archived as [[etherpadbackup:wmh2023-Toolhub,_Toolhunt,_and_the_quest]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/wmh2023-Overview_of_technical_areas_%26_projects]] was archived as [[etherpadbackup:wmh2023-Overview_of_technical_areas_%26_projects]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] was archived as [[etherpadbackup:wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] by [[User:BryanDavis|BryanDavis]]
* 00:23 [[etherpad:p/wmh2023-Best_practices_&amp;_ideas_for_implementing]] was archived as [[etherpadbackup:wmh2023-Best_practices_&amp;_ideas_for_implementing]] by [[User:BryanDavis|BryanDavis]]
* 00:23 [[etherpad:p/wmf11and12]] was archived as [[etherpadbackup:wmf11and12]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/wmcslightning]] was archived as [[etherpadbackup:wmcslightning]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/wmcs_nfs_future]] was archived as [[etherpadbackup:wmcs_nfs_future]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/wmcs-vs-rowb-upgrade]] was archived as [[etherpadbackup:wmcs-vs-rowb-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/wmcs-goals-q4-16-17]] was archived as [[etherpadbackup:wmcs-goals-q4-16-17]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/wmcs-goals-q2-17-18]] was archived as [[etherpadbackup:wmcs-goals-q2-17-18]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/wmcs-fallout]] was archived as [[etherpadbackup:wmcs-fallout]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/wmcs-cumin-fail]] was archived as [[etherpadbackup:wmcs-cumin-fail]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/wmcs-contributors]] was archived as [[etherpadbackup:wmcs-contributors]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/wmcs-2023-06-07]] was archived as [[etherpadbackup:wmcs-2023-06-07]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/wmconf2013-evaluating_programs]] was archived as [[etherpadbackup:wmconf2013-evaluating_programs]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/wmconf2013-evaluating-programs]] was archived as [[etherpadbackup:wmconf2013-evaluating-programs]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/wlmapp-iteration1-story-estimation-and-review]] was archived as [[etherpadbackup:wlmapp-iteration1-story-estimation-and-review]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/wlm-sprint5-retro]] was archived as [[etherpadbackup:wlm-sprint5-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/wlm-sprint4-retro]] was archived as [[etherpadbackup:wlm-sprint4-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/wlm-sprint3-retro]] was archived as [[etherpadbackup:wlm-sprint3-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/wlm-sprint2-retro]] was archived as [[etherpadbackup:wlm-sprint2-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:14 [[etherpad:p/wlm-sprint1-retro]] was archived as [[etherpadbackup:wlm-sprint1-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/wlm-project-retro]] was archived as [[etherpadbackup:wlm-project-retro]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/wlm-on-labs-steps]] was archived as [[etherpadbackup:wlm-on-labs-steps]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/wlm-api-admin-tree]] was archived as [[etherpadbackup:wlm-api-admin-tree]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/wishathon-march-2024-wishlist-overview-session]] was archived as [[etherpadbackup:wishathon-march-2024-wishlist-overview-session]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/wishathon-march-2024-showcase]] was archived as [[etherpadbackup:wishathon-march-2024-showcase]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/wishathon-march-2024-office-hour]] was archived as [[etherpadbackup:wishathon-march-2024-office-hour]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/wishathon-march-2024-fotw-session]] was archived as [[etherpadbackup:wishathon-march-2024-fotw-session]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/wikitechrebuild]] was archived as [[etherpadbackup:wikitechrebuild]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/wikitech-extensions]] was archived as [[etherpadbackup:wikitech-extensions]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/wikireplica-dns-switch]] was archived as [[etherpadbackup:wikireplica-dns-switch]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/wikireplica-beta]] was archived as [[etherpadbackup:wikireplica-beta]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/wikipedia-mobile-1-2-beta1-notes]] was archived as [[etherpadbackup:wikipedia-mobile-1-2-beta1-notes]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/wikipedia-ios-4.0.5-release-notes]] was archived as [[etherpadbackup:wikipedia-ios-4.0.5-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/wikipedia-app-saved-pages-architecture]] was archived as [[etherpadbackup:wikipedia-app-saved-pages-architecture]] by [[User:BryanDavis|BryanDavis]]
* 00:06 [[etherpad:p/wikipedia-app-ios-issues]] was archived as [[etherpadbackup:wikipedia-app-ios-issues]] by [[User:BryanDavis|BryanDavis]]
* 00:06 [[etherpad:p/wikimania_knowledge]] was archived as [[etherpadbackup:wikimania_knowledge]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/wikigroknextsteps]] was archived as [[etherpadbackup:wikigroknextsteps]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/wikigrokcollections]] was archived as [[etherpadbackup:wikigrokcollections]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/wikidev17-devwishlist]] was archived as [[etherpadbackup:wikidev17-devwishlist]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/wikidata_query_service_checkin]] was archived as [[etherpadbackup:wikidata_query_service_checkin]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/wikibits-undefault]] was archived as [[etherpadbackup:wikibits-undefault]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/wiki_technology_notes]] was archived as [[etherpadbackup:wiki_technology_notes]] by [[User:BryanDavis|BryanDavis]]
* 00:02 [[etherpad:p/widgets-for-uploadwizard]] was archived as [[etherpadbackup:widgets-for-uploadwizard]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/whm2023-Overview_of_technical_areas_&amp;_projects]] was archived as [[etherpadbackup:whm2023-Overview_of_technical_areas_&amp;_projects]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/whats-left-in-toolforge-eqiad]] was archived as [[etherpadbackup:whats-left-in-toolforge-eqiad]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/what_is_left_in_tampa]] was archived as [[etherpadbackup:what_is_left_in_tampa]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/webservice-k8s-announce]] was archived as [[etherpadbackup:webservice-k8s-announce]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-06 ===
* 23:59 [[etherpad:p/webrequest-2021-Q4-loss]] was archived as [[etherpadbackup:webrequest-2021-Q4-loss]] by [[User:BryanDavis|BryanDavis]]
* 23:59 [[etherpad:p/webreq_update]] was archived as [[etherpadbackup:webreq_update]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/we-really-love-nfs]] was archived as [[etherpadbackup:we-really-love-nfs]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/wdqs_graph_split_production_deploy]] was archived as [[etherpadbackup:wdqs_graph_split_production_deploy]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/wdqs-split-graph-rollout-plan]] was archived as [[etherpadbackup:wdqs-split-graph-rollout-plan]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/wdqs-2024-08-08]] was archived as [[etherpadbackup:wdqs-2024-08-08]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/wdio]] was archived as [[etherpadbackup:wdio]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/wcna_tools]] was archived as [[etherpadbackup:wcna_tools]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/wbq_test]] was archived as [[etherpadbackup:wbq_test]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/watchlist_id]] was archived as [[etherpadbackup:watchlist_id]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/wallaby-dev-magnum-install]] was archived as [[etherpadbackup:wallaby-dev-magnum-install]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/volans-tmp]] was archived as [[etherpadbackup:volans-tmp]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/vk-jumbo-cleanup]] was archived as [[etherpadbackup:vk-jumbo-cleanup]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/virt1006migrate]] was archived as [[etherpadbackup:virt1006migrate]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/vikassy]] was archived as [[etherpadbackup:vikassy]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/vikas_ve]] was archived as [[etherpadbackup:vikas_ve]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/vikas_vagrant]] was archived as [[etherpadbackup:vikas_vagrant]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/vikas-upload-error]] was archived as [[etherpadbackup:vikas-upload-error]] by [[User:BryanDavis|BryanDavis]]
* 23:50 [[etherpad:p/vikas-ruby-style]] was archived as [[etherpadbackup:vikas-ruby-style]] by [[User:BryanDavis|BryanDavis]]
* 23:49 [[etherpad:p/vikas-gerrit]] was archived as [[etherpadbackup:vikas-gerrit]] by [[User:BryanDavis|BryanDavis]]
* 23:49 [[etherpad:p/vikas]] was archived as [[etherpadbackup:vikas]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/ve-future-thoughts-2015]] was archived as [[etherpadbackup:ve-future-thoughts-2015]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/ve-2016-05-05]] was archived as [[etherpadbackup:ve-2016-05-05]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/ve]] was archived as [[etherpadbackup:ve]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/varnishkafka_troubleshooting]] was archived as [[etherpadbackup:varnishkafka_troubleshooting]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/vandrew]] was archived as [[etherpadbackup:vandrew]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/uw-cucumber-sketch]] was archived as [[etherpadbackup:uw-cucumber-sketch]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/uw-bugday-2014-09-09]] was archived as [[etherpadbackup:uw-bugday-2014-09-09]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/uploadwizard-refactor-planning]] was archived as [[etherpadbackup:uploadwizard-refactor-planning]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/uploadwizard-browser-tests]] was archived as [[etherpadbackup:uploadwizard-browser-tests]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/uploadcampaigns-squids]] was archived as [[etherpadbackup:uploadcampaigns-squids]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/upload-tool-class-structure]] was archived as [[etherpadbackup:upload-tool-class-structure]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/unsophabricated]] was archived as [[etherpadbackup:unsophabricated]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/unittestingbrownbag]] was archived as [[etherpadbackup:unittestingbrownbag]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/unified]] was archived as [[etherpadbackup:unified]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/unaliased]] was archived as [[etherpadbackup:unaliased]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/unable_to_move_translated_page]] was archived as [[etherpadbackup:unable_to_move_translated_page]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/twocolconflictdeploy]] was archived as [[etherpadbackup:twocolconflictdeploy]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/twiterbot-debuging]] was archived as [[etherpadbackup:twiterbot-debuging]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/trusty]] was archived as [[etherpadbackup:trusty]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/trixie-announcement]] was archived as [[etherpadbackup:trixie-announcement]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/trend-prod]] was archived as [[etherpadbackup:trend-prod]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/trebuchet-propose]] was archived as [[etherpadbackup:trebuchet-propose]] by [[User:BryanDavis|BryanDavis]]
* 23:37 [[etherpad:p/toomanythings]] was archived as [[etherpadbackup:toomanythings]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/toolssupport]] was archived as [[etherpadbackup:toolssupport]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/toolsdb-upgrade]] was archived as [[etherpadbackup:toolsdb-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/toolsdb-10.6]] was archived as [[etherpadbackup:toolsdb-10.6]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/tools-trusty-move]] was archived as [[etherpadbackup:tools-trusty-move]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/tools-the-great-recompress]] was archived as [[etherpadbackup:tools-the-great-recompress]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/tools-reboots-cve-0728]] was archived as [[etherpadbackup:tools-reboots-cve-0728]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/tools-reboots]] was archived as [[etherpadbackup:tools-reboots]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/tools-puppet-prefixes]] was archived as [[etherpadbackup:tools-puppet-prefixes]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/tools-migration-nodes]] was archived as [[etherpadbackup:tools-migration-nodes]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/tools-migration]] was archived as [[etherpadbackup:tools-migration]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/tools-kubernetes-scenarios]] was archived as [[etherpadbackup:tools-kubernetes-scenarios]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/tools-k8s-chat-01]] was archived as [[etherpadbackup:tools-k8s-chat-01]] by [[User:BryanDavis|BryanDavis]]
* 23:30 [[etherpad:p/toollabs20160127]] was archived as [[etherpadbackup:toollabs20160127]] by [[User:BryanDavis|BryanDavis]]
* 23:30 [[etherpad:p/toollabs-webservice-trusty-switch]] was archived as [[etherpadbackup:toollabs-webservice-trusty-switch]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/toollabs-uwsgi]] was archived as [[etherpadbackup:toollabs-uwsgi]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/toollabs-quarter-goal-q4-2014-15]] was archived as [[etherpadbackup:toollabs-quarter-goal-q4-2014-15]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/toollabs-impact]] was archived as [[etherpadbackup:toollabs-impact]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/toollabs-ideal-getting-started]] was archived as [[etherpadbackup:toollabs-ideal-getting-started]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/toollabs-cdnjs]] was archived as [[etherpadbackup:toollabs-cdnjs]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/toolforge-reboots]] was archived as [[etherpadbackup:toolforge-reboots]] by [[User:BryanDavis|BryanDavis]]
* 23:26 [[etherpad:p/toolforge-php72-webservice]] was archived as [[etherpadbackup:toolforge-php72-webservice]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/toolforge-k8s-upgrade-1.23]] was archived as [[etherpadbackup:toolforge-k8s-upgrade-1.23]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/toolforge-k8s-1.31]] was archived as [[etherpadbackup:toolforge-k8s-1.31]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/toolforge-k8s-1.28-to-1.29-upgrade]] was archived as [[etherpadbackup:toolforge-k8s-1.28-to-1.29-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/toolforge-2021-12-14]] was archived as [[etherpadbackup:toolforge-2021-12-14]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/tool_labs-doc]] was archived as [[etherpadbackup:tool_labs-doc]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/tonythomas01]] was archived as [[etherpadbackup:tonythomas01]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/tofu]] was archived as [[etherpadbackup:tofu]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/todays-regressions]] was archived as [[etherpadbackup:todays-regressions]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/title_value]] was archived as [[etherpadbackup:title_value]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/things_to_restart_after_rabbit_explosion]] was archived as [[etherpadbackup:things_to_restart_after_rabbit_explosion]] by [[User:BryanDavis|BryanDavis]]
* 06:03 [[etherpad:p/test]] was archived as [[etherpadbackup:test]] by [[User:BryanDavis|BryanDavis]]
* 06:03 [[etherpad:p/termdata]] was archived as [[etherpadbackup:termdata]] by [[User:BryanDavis|BryanDavis]]
* 06:02 [[etherpad:p/temp]] was archived as [[etherpadbackup:temp]] by [[User:BryanDavis|BryanDavis]]
* 06:02 [[etherpad:p/teach-yuvi-to-walk]] was archived as [[etherpadbackup:teach-yuvi-to-walk]] by [[User:BryanDavis|BryanDavis]]
* 06:01 [[etherpad:p/tcinterviewquestions]] was archived as [[etherpadbackup:tcinterviewquestions]] by [[User:BryanDavis|BryanDavis]]
* 06:00 [[etherpad:p/tallinntransportation]] was archived as [[etherpadbackup:tallinntransportation]] by [[User:BryanDavis|BryanDavis]]
* 06:00 [[etherpad:p/tabsinpuppet]] was archived as [[etherpadbackup:tabsinpuppet]] by [[User:BryanDavis|BryanDavis]]
* 05:59 [[etherpad:p/tFyLAurf0Z]] was archived as [[etherpadbackup:tFyLAurf0Z]] by [[User:BryanDavis|BryanDavis]]
* 05:59 [[etherpad:p/switchover]] was archived as [[etherpadbackup:switchover]] by [[User:BryanDavis|BryanDavis]]
* 05:58 [[etherpad:p/swiftfail]] was archived as [[etherpadbackup:swiftfail]] by [[User:BryanDavis|BryanDavis]]
* 05:58 [[etherpad:p/swhYPuS04T]] was archived as [[etherpadbackup:swhYPuS04T]] by [[User:BryanDavis|BryanDavis]]
* 05:57 [[etherpad:p/supersetdeprecation]] was archived as [[etherpadbackup:supersetdeprecation]] by [[User:BryanDavis|BryanDavis]]
* 05:57 [[etherpad:p/sulf]] was archived as [[etherpadbackup:sulf]] by [[User:BryanDavis|BryanDavis]]
* 05:56 [[etherpad:p/styleguide]] was archived as [[etherpadbackup:styleguide]] by [[User:BryanDavis|BryanDavis]]
* 05:56 [[etherpad:p/stuffs]] was archived as [[etherpadbackup:stuffs]] by [[User:BryanDavis|BryanDavis]]
* 05:55 [[etherpad:p/stubsExpansion]] was archived as [[etherpadbackup:stubsExpansion]] by [[User:BryanDavis|BryanDavis]]
* 05:55 [[etherpad:p/striker-prod-deploy]] was archived as [[etherpadbackup:striker-prod-deploy]] by [[User:BryanDavis|BryanDavis]]
* 05:54 [[etherpad:p/striker-gitlab-20220906]] was archived as [[etherpadbackup:striker-gitlab-20220906]] by [[User:BryanDavis|BryanDavis]]
* 05:54 [[etherpad:p/striker]] was archived as [[etherpadbackup:striker]] by [[User:BryanDavis|BryanDavis]]
* 05:53 [[etherpad:p/streaming_updater_cutover]] was archived as [[etherpadbackup:streaming_updater_cutover]] by [[User:BryanDavis|BryanDavis]]
* 05:53 [[etherpad:p/streaming-wdqs]] was archived as [[etherpadbackup:streaming-wdqs]] by [[User:BryanDavis|BryanDavis]]
* 05:52 [[etherpad:p/stillnopms]] was archived as [[etherpadbackup:stillnopms]] by [[User:BryanDavis|BryanDavis]]
* 05:52 [[etherpad:p/state_of_union_-_if_-network_bullet_points]] was archived as [[etherpadbackup:state_of_union_-_if_-network_bullet_points]] by [[User:BryanDavis|BryanDavis]]
* 05:51 [[etherpad:p/stat1_accounts]] was archived as [[etherpadbackup:stat1_accounts]] by [[User:BryanDavis|BryanDavis]]
* 05:51 [[etherpad:p/stat-analytics-vlan]] was archived as [[etherpadbackup:stat-analytics-vlan]] by [[User:BryanDavis|BryanDavis]]
* 05:50 [[etherpad:p/standalone_masters]] was archived as [[etherpadbackup:standalone_masters]] by [[User:BryanDavis|BryanDavis]]
* 05:50 [[etherpad:p/ssl]] was archived as [[etherpadbackup:ssl]] by [[User:BryanDavis|BryanDavis]]
* 05:49 [[etherpad:p/ssh-key-change-trixie]] was archived as [[etherpadbackup:ssh-key-change-trixie]] by [[User:BryanDavis|BryanDavis]]
* 05:48 [[etherpad:p/spark_pair_coding]] was archived as [[etherpadbackup:spark_pair_coding]] by [[User:BryanDavis|BryanDavis]]
* 05:48 [[etherpad:p/spark2-planning]] was archived as [[etherpadbackup:spark2-planning]] by [[User:BryanDavis|BryanDavis]]
* 05:47 [[etherpad:p/so1920.cs.unibo.it]] was archived as [[etherpadbackup:so1920.cs.unibo.it]] by [[User:BryanDavis|BryanDavis]]
* 05:47 [[etherpad:p/sistersearch-AB-test]] was archived as [[etherpadbackup:sistersearch-AB-test]] by [[User:BryanDavis|BryanDavis]]
* 05:46 [[etherpad:p/single-edit-tab]] was archived as [[etherpadbackup:single-edit-tab]] by [[User:BryanDavis|BryanDavis]]
* 05:46 [[etherpad:p/shit_tim_says]] was archived as [[etherpadbackup:shit_tim_says]] by [[User:BryanDavis|BryanDavis]]
* 05:45 [[etherpad:p/setting_up_ORES_lab_cluster]] was archived as [[etherpadbackup:setting_up_ORES_lab_cluster]] by [[User:BryanDavis|BryanDavis]]
* 05:45 [[etherpad:p/sendBeacon-2014-10-03]] was archived as [[etherpadbackup:sendBeacon-2014-10-03]] by [[User:BryanDavis|BryanDavis]]
* 05:44 [[etherpad:p/selenium]] was archived as [[etherpadbackup:selenium]] by [[User:BryanDavis|BryanDavis]]
* 05:44 [[etherpad:p/security_group_tests]] was archived as [[etherpadbackup:security_group_tests]] by [[User:BryanDavis|BryanDavis]]
* 05:43 [[etherpad:p/search_enhancements]] was archived as [[etherpadbackup:search_enhancements]] by [[User:BryanDavis|BryanDavis]]
* 05:43 [[etherpad:p/search-logging]] was archived as [[etherpadbackup:search-logging]] by [[User:BryanDavis|BryanDavis]]
* 05:42 [[etherpad:p/search-hypotesis-24-25]] was archived as [[etherpadbackup:search-hypotesis-24-25]] by [[User:BryanDavis|BryanDavis]]
* 05:42 [[etherpad:p/search-503s]] was archived as [[etherpadbackup:search-503s]] by [[User:BryanDavis|BryanDavis]]
* 05:41 [[etherpad:p/scrum-of-scrums]] was archived as [[etherpadbackup:scrum-of-scrums]] by [[User:BryanDavis|BryanDavis]]
* 05:41 [[etherpad:p/scoping-troubles]] was archived as [[etherpadbackup:scoping-troubles]] by [[User:BryanDavis|BryanDavis]]
* 05:40 [[etherpad:p/scap-commands]] was archived as [[etherpadbackup:scap-commands]] by [[User:BryanDavis|BryanDavis]]
* 05:40 [[etherpad:p/scap-201617-q2]] was archived as [[etherpadbackup:scap-201617-q2]] by [[User:BryanDavis|BryanDavis]]
* 05:39 [[etherpad:p/rubocop-2015-02-10]] was archived as [[etherpadbackup:rubocop-2015-02-10]] by [[User:BryanDavis|BryanDavis]]
* 05:39 [[etherpad:p/rsyncnova]] was archived as [[etherpadbackup:rsyncnova]] by [[User:BryanDavis|BryanDavis]]
* 05:38 [[etherpad:p/rollout-ldap-block-change]] was archived as [[etherpadbackup:rollout-ldap-block-change]] by [[User:BryanDavis|BryanDavis]]
* 05:38 [[etherpad:p/role-mediainfo]] was archived as [[etherpadbackup:role-mediainfo]] by [[User:BryanDavis|BryanDavis]]
* 05:37 [[etherpad:p/rockyupgrade]] was archived as [[etherpadbackup:rockyupgrade]] by [[User:BryanDavis|BryanDavis]]
* 05:37 [[etherpad:p/robots.txt]] was archived as [[etherpadbackup:robots.txt]] by [[User:BryanDavis|BryanDavis]]
* 05:36 [[etherpad:p/rmrebootsforcephmons]] was archived as [[etherpadbackup:rmrebootsforcephmons]] by [[User:BryanDavis|BryanDavis]]
* 05:35 [[etherpad:p/revscoring_user_oriented]] was archived as [[etherpadbackup:revscoring_user_oriented]] by [[User:BryanDavis|BryanDavis]]
* 05:35 [[etherpad:p/revscoring_process]] was archived as [[etherpadbackup:revscoring_process]] by [[User:BryanDavis|BryanDavis]]
* 05:34 [[etherpad:p/revscoring_languages]] was archived as [[etherpadbackup:revscoring_languages]] by [[User:BryanDavis|BryanDavis]]
* 05:34 [[etherpad:p/revscoring_hackathon]] was archived as [[etherpadbackup:revscoring_hackathon]] by [[User:BryanDavis|BryanDavis]]
* 05:33 [[etherpad:p/review-workflow-yuvi]] was archived as [[etherpadbackup:review-workflow-yuvi]] by [[User:BryanDavis|BryanDavis]]
* 05:33 [[etherpad:p/resurrect_omega]] was archived as [[etherpadbackup:resurrect_omega]] by [[User:BryanDavis|BryanDavis]]
* 05:32 [[etherpad:p/resthack]] was archived as [[etherpadbackup:resthack]] by [[User:BryanDavis|BryanDavis]]
* 05:32 [[etherpad:p/restbase-parsoid-storage-rollout]] was archived as [[etherpadbackup:restbase-parsoid-storage-rollout]] by [[User:BryanDavis|BryanDavis]]
* 05:31 [[etherpad:p/researchers_in_our_midst]] was archived as [[etherpadbackup:researchers_in_our_midst]] by [[User:BryanDavis|BryanDavis]]
* 05:31 [[etherpad:p/research_showcase_dec14]] was archived as [[etherpadbackup:research_showcase_dec14]] by [[User:BryanDavis|BryanDavis]]
* 05:30 [[etherpad:p/replicas-temp-accounts]] was archived as [[etherpadbackup:replicas-temp-accounts]] by [[User:BryanDavis|BryanDavis]]
* 05:30 [[etherpad:p/rep-lag-s8]] was archived as [[etherpadbackup:rep-lag-s8]] by [[User:BryanDavis|BryanDavis]]
* 05:29 [[etherpad:p/remaining-ldap]] was archived as [[etherpadbackup:remaining-ldap]] by [[User:BryanDavis|BryanDavis]]
* 05:29 [[etherpad:p/refinery_deploy]] was archived as [[etherpadbackup:refinery_deploy]] by [[User:BryanDavis|BryanDavis]]
* 05:28 [[etherpad:p/red-links]] was archived as [[etherpadbackup:red-links]] by [[User:BryanDavis|BryanDavis]]
* 05:28 [[etherpad:p/realm_networks]] was archived as [[etherpadbackup:realm_networks]] by [[User:BryanDavis|BryanDavis]]
* 05:27 [[etherpad:p/reading-infra-2015-16-q4]] was archived as [[etherpadbackup:reading-infra-2015-16-q4]] by [[User:BryanDavis|BryanDavis]]
* 05:27 [[etherpad:p/rdf-flink-k8s]] was archived as [[etherpadbackup:rdf-flink-k8s]] by [[User:BryanDavis|BryanDavis]]
* 05:26 [[etherpad:p/rdbmess-2021]] was archived as [[etherpadbackup:rdbmess-2021]] by [[User:BryanDavis|BryanDavis]]
* 05:26 [[etherpad:p/rcstream-misc]] was archived as [[etherpadbackup:rcstream-misc]] by [[User:BryanDavis|BryanDavis]]
* 05:25 [[etherpad:p/rc1-release-email]] was archived as [[etherpadbackup:rc1-release-email]] by [[User:BryanDavis|BryanDavis]]
* 05:25 [[etherpad:p/rc.2-announce]] was archived as [[etherpadbackup:rc.2-announce]] by [[User:BryanDavis|BryanDavis]]
* 05:24 [[etherpad:p/rar]] was archived as [[etherpadbackup:rar]] by [[User:BryanDavis|BryanDavis]]
* 05:24 [[etherpad:p/rada12dubna]] was archived as [[etherpadbackup:rada12dubna]] by [[User:BryanDavis|BryanDavis]]
* 05:23 [[etherpad:p/r.d2e7710af5a20eefd93babda3f38929a]] was archived as [[etherpadbackup:r.d2e7710af5a20eefd93babda3f38929a]] by [[User:BryanDavis|BryanDavis]]
* 05:22 [[etherpad:p/r.B0XlRhOdKRWT6xuH]] was archived as [[etherpadbackup:r.B0XlRhOdKRWT6xuH]] by [[User:BryanDavis|BryanDavis]]
* 05:22 [[etherpad:p/quicksurveys]] was archived as [[etherpadbackup:quicksurveys]] by [[User:BryanDavis|BryanDavis]]
* 05:21 [[etherpad:p/quibble-0.0.36]] was archived as [[etherpadbackup:quibble-0.0.36]] by [[User:BryanDavis|BryanDavis]]
* 05:21 [[etherpad:p/quarry-wdqs-integration]] was archived as [[etherpadbackup:quarry-wdqs-integration]] by [[User:BryanDavis|BryanDavis]]
* 05:20 [[etherpad:p/quarry-for-pwb-scripts]] was archived as [[etherpadbackup:quarry-for-pwb-scripts]] by [[User:BryanDavis|BryanDavis]]
* 05:20 [[etherpad:p/quarry-announce-email]] was archived as [[etherpadbackup:quarry-announce-email]] by [[User:BryanDavis|BryanDavis]]
* 05:19 [[etherpad:p/q_planning_retro]] was archived as [[etherpadbackup:q_planning_retro]] by [[User:BryanDavis|BryanDavis]]
* 05:19 [[etherpad:p/q4planningretro]] was archived as [[etherpadbackup:q4planningretro]] by [[User:BryanDavis|BryanDavis]]
* 05:18 [[etherpad:p/q4planningnotes]] was archived as [[etherpadbackup:q4planningnotes]] by [[User:BryanDavis|BryanDavis]]
* 05:18 [[etherpad:p/q4_checkin]] was archived as [[etherpadbackup:q4_checkin]] by [[User:BryanDavis|BryanDavis]]
* 05:17 [[etherpad:p/q2_eng_goal]] was archived as [[etherpadbackup:q2_eng_goal]] by [[User:BryanDavis|BryanDavis]]
* 05:17 [[etherpad:p/python_mediawiki_devs]] was archived as [[etherpadbackup:python_mediawiki_devs]] by [[User:BryanDavis|BryanDavis]]
* 05:16 [[etherpad:p/python]] was archived as [[etherpadbackup:python]] by [[User:BryanDavis|BryanDavis]]
* 05:16 [[etherpad:p/push-to-deploy-beta-announce]] was archived as [[etherpadbackup:push-to-deploy-beta-announce]] by [[User:BryanDavis|BryanDavis]]
* 05:15 [[etherpad:p/puppetswat]] was archived as [[etherpadbackup:puppetswat]] by [[User:BryanDavis|BryanDavis]]
* 05:15 [[etherpad:p/puppetmaster-enc]] was archived as [[etherpadbackup:puppetmaster-enc]] by [[User:BryanDavis|BryanDavis]]
* 05:14 [[etherpad:p/puppet_upgrade_vms]] was archived as [[etherpadbackup:puppet_upgrade_vms]] by [[User:BryanDavis|BryanDavis]]
* 05:14 [[etherpad:p/puppet3]] was archived as [[etherpadbackup:puppet3]] by [[User:BryanDavis|BryanDavis]]
* 05:13 [[etherpad:p/publish-new-mobile-report]] was archived as [[etherpadbackup:publish-new-mobile-report]] by [[User:BryanDavis|BryanDavis]]
* 05:13 [[etherpad:p/proxy-buster]] was archived as [[etherpadbackup:proxy-buster]] by [[User:BryanDavis|BryanDavis]]
* 05:12 [[etherpad:p/preciseonlabs]] was archived as [[etherpadbackup:preciseonlabs]] by [[User:BryanDavis|BryanDavis]]
* 05:12 [[etherpad:p/precise-tools]] was archived as [[etherpadbackup:precise-tools]] by [[User:BryanDavis|BryanDavis]]
* 05:11 [[etherpad:p/poem]] was archived as [[etherpadbackup:poem]] by [[User:BryanDavis|BryanDavis]]
* 05:11 [[etherpad:p/play_store_what's_new]] was archived as [[etherpadbackup:play_store_what's_new]] by [[User:BryanDavis|BryanDavis]]
* 05:10 [[etherpad:p/pii_pageviews]] was archived as [[etherpadbackup:pii_pageviews]] by [[User:BryanDavis|BryanDavis]]
* 05:09 [[etherpad:p/phab_roles]] was archived as [[etherpadbackup:phab_roles]] by [[User:BryanDavis|BryanDavis]]
* 05:09 [[etherpad:p/phab-2016-02-03]] was archived as [[etherpadbackup:phab-2016-02-03]] by [[User:BryanDavis|BryanDavis]]
* 05:08 [[etherpad:p/phab]] was archived as [[etherpadbackup:phab]] by [[User:BryanDavis|BryanDavis]]
* 05:08 [[etherpad:p/performance-team-weekly-update]] was archived as [[etherpadbackup:performance-team-weekly-update]] by [[User:BryanDavis|BryanDavis]]
* 05:07 [[etherpad:p/performance-k8s]] was archived as [[etherpadbackup:performance-k8s]] by [[User:BryanDavis|BryanDavis]]
* 05:07 [[etherpad:p/perfnotice]] was archived as [[etherpadbackup:perfnotice]] by [[User:BryanDavis|BryanDavis]]
* 05:06 [[etherpad:p/perf-announcements]] was archived as [[etherpadbackup:perf-announcements]] by [[User:BryanDavis|BryanDavis]]
* 05:06 [[etherpad:p/perf-20160223]] was archived as [[etherpadbackup:perf-20160223]] by [[User:BryanDavis|BryanDavis]]
* 05:05 [[etherpad:p/paws-tools]] was archived as [[etherpadbackup:paws-tools]] by [[User:BryanDavis|BryanDavis]]
* 05:05 [[etherpad:p/paws-public-url-structure]] was archived as [[etherpadbackup:paws-public-url-structure]] by [[User:BryanDavis|BryanDavis]]
* 05:04 [[etherpad:p/paste]] was archived as [[etherpadbackup:paste]] by [[User:BryanDavis|BryanDavis]]
* 05:04 [[etherpad:p/parsoid_docs]] was archived as [[etherpadbackup:parsoid_docs]] by [[User:BryanDavis|BryanDavis]]
* 05:03 [[etherpad:p/parsoid-2019-01-16]] was archived as [[etherpadbackup:parsoid-2019-01-16]] by [[User:BryanDavis|BryanDavis]]
* 05:03 [[etherpad:p/parsing_team_meeting_archive]] was archived as [[etherpadbackup:parsing_team_meeting_archive]] by [[User:BryanDavis|BryanDavis]]
* 05:02 [[etherpad:p/parser-api-2018]] was archived as [[etherpadbackup:parser-api-2018]] by [[User:BryanDavis|BryanDavis]]
* 05:02 [[etherpad:p/pageview_api_nodes]] was archived as [[etherpadbackup:pageview_api_nodes]] by [[User:BryanDavis|BryanDavis]]
* 05:01 [[etherpad:p/pages-needing-investigation]] was archived as [[etherpadbackup:pages-needing-investigation]] by [[User:BryanDavis|BryanDavis]]
* 05:01 [[etherpad:p/page-styling-zurich]] was archived as [[etherpadbackup:page-styling-zurich]] by [[User:BryanDavis|BryanDavis]]
* 05:00 [[etherpad:p/otrs-migration]] was archived as [[etherpadbackup:otrs-migration]] by [[User:BryanDavis|BryanDavis]]
* 05:00 [[etherpad:p/ores_task_tracking]] was archived as [[etherpadbackup:ores_task_tracking]] by [[User:BryanDavis|BryanDavis]]
* 04:59 [[etherpad:p/ores]] was archived as [[etherpadbackup:ores]] by [[User:BryanDavis|BryanDavis]]
* 04:58 [[etherpad:p/ops-offsite-discussions]] was archived as [[etherpadbackup:ops-offsite-discussions]] by [[User:BryanDavis|BryanDavis]]
* 04:58 [[etherpad:p/ops-offsite-2016-lightning-talks]] was archived as [[etherpadbackup:ops-offsite-2016-lightning-talks]] by [[User:BryanDavis|BryanDavis]]
* 04:57 [[etherpad:p/ops-offsite-2016-labstore-future]] was archived as [[etherpadbackup:ops-offsite-2016-labstore-future]] by [[User:BryanDavis|BryanDavis]]
* 04:57 [[etherpad:p/opensearch-opportunities-for-improvement]] was archived as [[etherpadbackup:opensearch-opportunities-for-improvement]] by [[User:BryanDavis|BryanDavis]]
* 04:56 [[etherpad:p/opendj-migration]] was archived as [[etherpadbackup:opendj-migration]] by [[User:BryanDavis|BryanDavis]]
* 04:56 [[etherpad:p/open_infra_workshop]] was archived as [[etherpadbackup:open_infra_workshop]] by [[User:BryanDavis|BryanDavis]]
* 04:55 [[etherpad:p/opcache-💩-2020-10-01]] was archived as [[etherpadbackup:opcache-💩-2020-10-01]] by [[User:BryanDavis|BryanDavis]]
* 04:55 [[etherpad:p/oozie-restart]] was archived as [[etherpadbackup:oozie-restart]] by [[User:BryanDavis|BryanDavis]]
* 04:54 [[etherpad:p/oozie]] was archived as [[etherpadbackup:oozie]] by [[User:BryanDavis|BryanDavis]]
* 04:54 [[etherpad:p/omega-restarts]] was archived as [[etherpadbackup:omega-restarts]] by [[User:BryanDavis|BryanDavis]]
* 04:53 [[etherpad:p/oldbastion]] was archived as [[etherpadbackup:oldbastion]] by [[User:BryanDavis|BryanDavis]]
* 04:53 [[etherpad:p/okapi-dumps-2021-02-01]] was archived as [[etherpadbackup:okapi-dumps-2021-02-01]] by [[User:BryanDavis|BryanDavis]]
* 04:52 [[etherpad:p/ocata-upgrade]] was archived as [[etherpadbackup:ocata-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 04:52 [[etherpad:p/o1Ub87jizX]] was archived as [[etherpadbackup:o1Ub87jizX]] by [[User:BryanDavis|BryanDavis]]
* 04:51 [[etherpad:p/o]] was archived as [[etherpadbackup:o]] by [[User:BryanDavis|BryanDavis]]
* 04:51 [[etherpad:p/nuria]] was archived as [[etherpadbackup:nuria]] by [[User:BryanDavis|BryanDavis]]
* 04:50 [[etherpad:p/npp_discussion_wmhack17]] was archived as [[etherpadbackup:npp_discussion_wmhack17]] by [[User:BryanDavis|BryanDavis]]
* 04:50 [[etherpad:p/nodes_with_a_public_IP]] was archived as [[etherpadbackup:nodes_with_a_public_IP]] by [[User:BryanDavis|BryanDavis]]
* 04:49 [[etherpad:p/nodepool-migration]] was archived as [[etherpadbackup:nodepool-migration]] by [[User:BryanDavis|BryanDavis]]
* 04:49 [[etherpad:p/night-mode-colors]] was archived as [[etherpadbackup:night-mode-colors]] by [[User:BryanDavis|BryanDavis]]
* 04:48 [[etherpad:p/ng8EYRhQ7SIXy7cLBexZ]] was archived as [[etherpadbackup:ng8EYRhQ7SIXy7cLBexZ]] by [[User:BryanDavis|BryanDavis]]
* 04:48 [[etherpad:p/nfsprojectchange]] was archived as [[etherpadbackup:nfsprojectchange]] by [[User:BryanDavis|BryanDavis]]
* 04:47 [[etherpad:p/nfsmaint]] was archived as [[etherpadbackup:nfsmaint]] by [[User:BryanDavis|BryanDavis]]
* 04:47 [[etherpad:p/nfs]] was archived as [[etherpadbackup:nfs]] by [[User:BryanDavis|BryanDavis]]
* 04:46 [[etherpad:p/newton_upgrade_plan]] was archived as [[etherpadbackup:newton_upgrade_plan]] by [[User:BryanDavis|BryanDavis]]
* 04:45 [[etherpad:p/newsletters-contenthandler]] was archived as [[etherpadbackup:newsletters-contenthandler]] by [[User:BryanDavis|BryanDavis]]
* 04:45 [[etherpad:p/newpyter]] was archived as [[etherpadbackup:newpyter]] by [[User:BryanDavis|BryanDavis]]
* 04:44 [[etherpad:p/new-webnode-toollabs]] was archived as [[etherpadbackup:new-webnode-toollabs]] by [[User:BryanDavis|BryanDavis]]
* 04:44 [[etherpad:p/networkoutage]] was archived as [[etherpadbackup:networkoutage]] by [[User:BryanDavis|BryanDavis]]
* 04:43 [[etherpad:p/netbox4-upgrade]] was archived as [[etherpadbackup:netbox4-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 04:43 [[etherpad:p/netbox-fail]] was archived as [[etherpadbackup:netbox-fail]] by [[User:BryanDavis|BryanDavis]]
* 04:42 [[etherpad:p/naming-visualization]] was archived as [[etherpadbackup:naming-visualization]] by [[User:BryanDavis|BryanDavis]]
* 04:42 [[etherpad:p/nTbAb5yPvBFATUylV7Vj]] was archived as [[etherpadbackup:nTbAb5yPvBFATUylV7Vj]] by [[User:BryanDavis|BryanDavis]]
* 04:41 [[etherpad:p/mwtidy-checkerrors]] was archived as [[etherpadbackup:mwtidy-checkerrors]] by [[User:BryanDavis|BryanDavis]]
* 04:41 [[etherpad:p/mwds15-spec-oriented-architecture]] was archived as [[etherpadbackup:mwds15-spec-oriented-architecture]] by [[User:BryanDavis|BryanDavis]]
* 04:40 [[etherpad:p/mwds-profiling]] was archived as [[etherpadbackup:mwds-profiling]] by [[User:BryanDavis|BryanDavis]]
* 04:40 [[etherpad:p/mw1-25-jsdeprecate]] was archived as [[etherpadbackup:mw1-25-jsdeprecate]] by [[User:BryanDavis|BryanDavis]]
* 04:39 [[etherpad:p/mw-php70-tests]] was archived as [[etherpadbackup:mw-php70-tests]] by [[User:BryanDavis|BryanDavis]]
* 04:39 [[etherpad:p/mvrelease]] was archived as [[etherpadbackup:mvrelease]] by [[User:BryanDavis|BryanDavis]]
* 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-12-24]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-12-24]] by [[User:BryanDavis|BryanDavis]]
* 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-01-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-01-14]] by [[User:BryanDavis|BryanDavis]]
* 04:37 [[etherpad:p/multimedia-weekly-meeting-2014-01-06]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-01-06]] by [[User:BryanDavis|BryanDavis]]
* 04:37 [[etherpad:p/multimedia-mmv-technical-retrospective]] was archived as [[etherpadbackup:multimedia-mmv-technical-retrospective]] by [[User:BryanDavis|BryanDavis]]
* 04:36 [[etherpad:p/multimedia-design-05-15-2014]] was archived as [[etherpadbackup:multimedia-design-05-15-2014]] by [[User:BryanDavis|BryanDavis]]
* 04:36 [[etherpad:p/multimedia-analytics-queries]] was archived as [[etherpadbackup:multimedia-analytics-queries]] by [[User:BryanDavis|BryanDavis]]
* 04:35 [[etherpad:p/morethan10g]] was archived as [[etherpadbackup:morethan10g]] by [[User:BryanDavis|BryanDavis]]
* 04:35 [[etherpad:p/monitoring-lvs-l2]] was archived as [[etherpadbackup:monitoring-lvs-l2]] by [[User:BryanDavis|BryanDavis]]
* 04:34 [[etherpad:p/mod-conf]] was archived as [[etherpadbackup:mod-conf]] by [[User:BryanDavis|BryanDavis]]
* 04:34 [[etherpad:p/mobileweb-wsj]] was archived as [[etherpadbackup:mobileweb-wsj]] by [[User:BryanDavis|BryanDavis]]
* 04:33 [[etherpad:p/mobilesprintname]] was archived as [[etherpadbackup:mobilesprintname]] by [[User:BryanDavis|BryanDavis]]
* 04:33 [[etherpad:p/mobilenav-sprint2-retro]] was archived as [[etherpadbackup:mobilenav-sprint2-retro]] by [[User:BryanDavis|BryanDavis]]
* 04:32 [[etherpad:p/mobilenav-sprint1-retro]] was archived as [[etherpadbackup:mobilenav-sprint1-retro]] by [[User:BryanDavis|BryanDavis]]
* 04:32 [[etherpad:p/mobileappplanningQ32014]] was archived as [[etherpadbackup:mobileappplanningQ32014]] by [[User:BryanDavis|BryanDavis]]
* 04:31 [[etherpad:p/mobile_web_q3]] was archived as [[etherpadbackup:mobile_web_q3]] by [[User:BryanDavis|BryanDavis]]
* 04:30 [[etherpad:p/mobile_web_health_check]] was archived as [[etherpadbackup:mobile_web_health_check]] by [[User:BryanDavis|BryanDavis]]
* 04:30 [[etherpad:p/mobile_web_design_review]] was archived as [[etherpadbackup:mobile_web_design_review]] by [[User:BryanDavis|BryanDavis]]
* 04:29 [[etherpad:p/mobile_web_breakout]] was archived as [[etherpadbackup:mobile_web_breakout]] by [[User:BryanDavis|BryanDavis]]
* 04:29 [[etherpad:p/mobile_apps_onboarding_brian]] was archived as [[etherpadbackup:mobile_apps_onboarding_brian]] by [[User:BryanDavis|BryanDavis]]
* 04:28 [[etherpad:p/mobile-users-faq]] was archived as [[etherpadbackup:mobile-users-faq]] by [[User:BryanDavis|BryanDavis]]
* 04:28 [[etherpad:p/mobile-tutorial-practice]] was archived as [[etherpadbackup:mobile-tutorial-practice]] by [[User:BryanDavis|BryanDavis]]
* 04:27 [[etherpad:p/mobile-retrospective-q3-2013]] was archived as [[etherpadbackup:mobile-retrospective-q3-2013]] by [[User:BryanDavis|BryanDavis]]
* 04:27 [[etherpad:p/mobile-retrospective-20130128]] was archived as [[etherpadbackup:mobile-retrospective-20130128]] by [[User:BryanDavis|BryanDavis]]
* 04:26 [[etherpad:p/mobile-retrospective-20130114]] was archived as [[etherpadbackup:mobile-retrospective-20130114]] by [[User:BryanDavis|BryanDavis]]
* 04:26 [[etherpad:p/mobile-retrospective-20121217]] was archived as [[etherpadbackup:mobile-retrospective-20121217]] by [[User:BryanDavis|BryanDavis]]
* 04:25 [[etherpad:p/mobile-retrospective-20120102]] was archived as [[etherpadbackup:mobile-retrospective-20120102]] by [[User:BryanDavis|BryanDavis]]
* 04:25 [[etherpad:p/mobile-q4-planning-2013]] was archived as [[etherpadbackup:mobile-q4-planning-2013]] by [[User:BryanDavis|BryanDavis]]
* 04:24 [[etherpad:p/mobile-q3-planning-2013]] was archived as [[etherpadbackup:mobile-q3-planning-2013]] by [[User:BryanDavis|BryanDavis]]
* 04:24 [[etherpad:p/mobile-ops-syncup-28feb2013]] was archived as [[etherpadbackup:mobile-ops-syncup-28feb2013]] by [[User:BryanDavis|BryanDavis]]
* 04:23 [[etherpad:p/mobile-ime-support]] was archived as [[etherpadbackup:mobile-ime-support]] by [[User:BryanDavis|BryanDavis]]
* 04:23 [[etherpad:p/mobile-campaigns]] was archived as [[etherpadbackup:mobile-campaigns]] by [[User:BryanDavis|BryanDavis]]
* 04:22 [[etherpad:p/mobile-app-workflow-el-1]] was archived as [[etherpadbackup:mobile-app-workflow-el-1]] by [[User:BryanDavis|BryanDavis]]
* 04:22 [[etherpad:p/mobile-app-versions]] was archived as [[etherpadbackup:mobile-app-versions]] by [[User:BryanDavis|BryanDavis]]
* 04:21 [[etherpad:p/mobile-app-otrs-feedback]] was archived as [[etherpadbackup:mobile-app-otrs-feedback]] by [[User:BryanDavis|BryanDavis]]
* 04:21 [[etherpad:p/mobile-app-dashboard]] was archived as [[etherpadbackup:mobile-app-dashboard]] by [[User:BryanDavis|BryanDavis]]
* 04:20 [[etherpad:p/mmv-text-size-changes]] was archived as [[etherpadbackup:mmv-text-size-changes]] by [[User:BryanDavis|BryanDavis]]
* 04:20 [[etherpad:p/mmodell]] was archived as [[etherpadbackup:mmodell]] by [[User:BryanDavis|BryanDavis]]
* 04:19 [[etherpad:p/mm-standup-notes]] was archived as [[etherpadbackup:mm-standup-notes]] by [[User:BryanDavis|BryanDavis]]
* 04:19 [[etherpad:p/misc-migration]] was archived as [[etherpadbackup:misc-migration]] by [[User:BryanDavis|BryanDavis]]
* 04:18 [[etherpad:p/migrate-k8s-etcd]] was archived as [[etherpadbackup:migrate-k8s-etcd]] by [[User:BryanDavis|BryanDavis]]
* 04:18 [[etherpad:p/mf]] was archived as [[etherpadbackup:mf]] by [[User:BryanDavis|BryanDavis]]
* 04:17 [[etherpad:p/mediawiki-vagrant]] was archived as [[etherpadbackup:mediawiki-vagrant]] by [[User:BryanDavis|BryanDavis]]
* 04:16 [[etherpad:p/mazza]] was archived as [[etherpadbackup:mazza]] by [[User:BryanDavis|BryanDavis]]
* 04:16 [[etherpad:p/mayankmadan]] was archived as [[etherpadbackup:mayankmadan]] by [[User:BryanDavis|BryanDavis]]
* 04:15 [[etherpad:p/mailman-Aug-2015]] was archived as [[etherpadbackup:mailman-Aug-2015]] by [[User:BryanDavis|BryanDavis]]
* 04:15 [[etherpad:p/magru_server_swaps]] was archived as [[etherpadbackup:magru_server_swaps]] by [[User:BryanDavis|BryanDavis]]
* 04:14 [[etherpad:p/magnumusersfeb2026]] was archived as [[etherpadbackup:magnumusersfeb2026]] by [[User:BryanDavis|BryanDavis]]
* 04:14 [[etherpad:p/mBBgYyJeBG]] was archived as [[etherpadbackup:mBBgYyJeBG]] by [[User:BryanDavis|BryanDavis]]
* 04:13 [[etherpad:p/m1-service-owners]] was archived as [[etherpadbackup:m1-service-owners]] by [[User:BryanDavis|BryanDavis]]
* 04:13 [[etherpad:p/lyon-service-template]] was archived as [[etherpadbackup:lyon-service-template]] by [[User:BryanDavis|BryanDavis]]
* 04:12 [[etherpad:p/lucid_mini_hack]] was archived as [[etherpadbackup:lucid_mini_hack]] by [[User:BryanDavis|BryanDavis]]
* 04:12 [[etherpad:p/lubaochuan]] was archived as [[etherpadbackup:lubaochuan]] by [[User:BryanDavis|BryanDavis]]
* 04:11 [[etherpad:p/logchange]] was archived as [[etherpadbackup:logchange]] by [[User:BryanDavis|BryanDavis]]
* 04:11 [[etherpad:p/linkpass]] was archived as [[etherpadbackup:linkpass]] by [[User:BryanDavis|BryanDavis]]
* 04:10 [[etherpad:p/legal-text-upload-dialog]] was archived as [[etherpadbackup:legal-text-upload-dialog]] by [[User:BryanDavis|BryanDavis]]
* 04:10 [[etherpad:p/language-and-localization]] was archived as [[etherpadbackup:language-and-localization]] by [[User:BryanDavis|BryanDavis]]
* 04:09 [[etherpad:p/langaugescreenshot-upload-error]] was archived as [[etherpadbackup:langaugescreenshot-upload-error]] by [[User:BryanDavis|BryanDavis]]
* 04:09 [[etherpad:p/lang-bar]] was archived as [[etherpadbackup:lang-bar]] by [[User:BryanDavis|BryanDavis]]
* 04:08 [[etherpad:p/labwebneedsports]] was archived as [[etherpadbackup:labwebneedsports]] by [[User:BryanDavis|BryanDavis]]
* 04:08 [[etherpad:p/labweb]] was archived as [[etherpadbackup:labweb]] by [[User:BryanDavis|BryanDavis]]
* 04:07 [[etherpad:p/labvirt_reboots]] was archived as [[etherpadbackup:labvirt_reboots]] by [[User:BryanDavis|BryanDavis]]
* 04:07 [[etherpad:p/labvirt1015]] was archived as [[etherpadbackup:labvirt1015]] by [[User:BryanDavis|BryanDavis]]
* 04:06 [[etherpad:p/labvirt1003instances]] was archived as [[etherpadbackup:labvirt1003instances]] by [[User:BryanDavis|BryanDavis]]
* 04:06 [[etherpad:p/labtes_mitaka_upgrade_notes]] was archived as [[etherpadbackup:labtes_mitaka_upgrade_notes]] by [[User:BryanDavis|BryanDavis]]
* 04:05 [[etherpad:p/labstore10045-move]] was archived as [[etherpadbackup:labstore10045-move]] by [[User:BryanDavis|BryanDavis]]
* 04:04 [[etherpad:p/labservices_failover]] was archived as [[etherpadbackup:labservices_failover]] by [[User:BryanDavis|BryanDavis]]
* 04:04 [[etherpad:p/labsdb10067]] was archived as [[etherpadbackup:labsdb10067]] by [[User:BryanDavis|BryanDavis]]
* 04:03 [[etherpad:p/labsdb-replication]] was archived as [[etherpadbackup:labsdb-replication]] by [[User:BryanDavis|BryanDavis]]
* 04:03 [[etherpad:p/labsdb-goal]] was archived as [[etherpadbackup:labsdb-goal]] by [[User:BryanDavis|BryanDavis]]
* 04:02 [[etherpad:p/labsdb-chat-july-18]] was archived as [[etherpadbackup:labsdb-chat-july-18]] by [[User:BryanDavis|BryanDavis]]
* 04:02 [[etherpad:p/labs_migration]] was archived as [[etherpadbackup:labs_migration]] by [[User:BryanDavis|BryanDavis]]
* 04:01 [[etherpad:p/labs-survey-interview-questions]] was archived as [[etherpadbackup:labs-survey-interview-questions]] by [[User:BryanDavis|BryanDavis]]
* 04:01 [[etherpad:p/labs-report-q4-1]] was archived as [[etherpadbackup:labs-report-q4-1]] by [[User:BryanDavis|BryanDavis]]
* 04:00 [[etherpad:p/labs-releng-k8s-confluence]] was archived as [[etherpadbackup:labs-releng-k8s-confluence]] by [[User:BryanDavis|BryanDavis]]
* 04:00 [[etherpad:p/labs-nfs-20140413-nfsoutage]] was archived as [[etherpadbackup:labs-nfs-20140413-nfsoutage]] by [[User:BryanDavis|BryanDavis]]
* 03:59 [[etherpad:p/labs-migration]] was archived as [[etherpadbackup:labs-migration]] by [[User:BryanDavis|BryanDavis]]
* 03:59 [[etherpad:p/labs-meeting-apr-25-2016]] was archived as [[etherpadbackup:labs-meeting-apr-25-2016]] by [[User:BryanDavis|BryanDavis]]
* 03:58 [[etherpad:p/labs-incident-timeline]] was archived as [[etherpadbackup:labs-incident-timeline]] by [[User:BryanDavis|BryanDavis]]
* 03:58 [[etherpad:p/labs-cleanup-2015]] was archived as [[etherpadbackup:labs-cleanup-2015]] by [[User:BryanDavis|BryanDavis]]
* 03:57 [[etherpad:p/labpuppetmaster1001-stragglers]] was archived as [[etherpadbackup:labpuppetmaster1001-stragglers]] by [[User:BryanDavis|BryanDavis]]
* 03:57 [[etherpad:p/labnetfailover]] was archived as [[etherpadbackup:labnetfailover]] by [[User:BryanDavis|BryanDavis]]
* 03:56 [[etherpad:p/l10nupdate]] was archived as [[etherpadbackup:l10nupdate]] by [[User:BryanDavis|BryanDavis]]
* 03:56 [[etherpad:p/l10n-translate-centralnotice]] was archived as [[etherpadbackup:l10n-translate-centralnotice]] by [[User:BryanDavis|BryanDavis]]
* 03:55 [[etherpad:p/l10n-team-2013-10]] was archived as [[etherpadbackup:l10n-team-2013-10]] by [[User:BryanDavis|BryanDavis]]
* 03:55 [[etherpad:p/kubecon-2017-offsite-agenda]] was archived as [[etherpadbackup:kubecon-2017-offsite-agenda]] by [[User:BryanDavis|BryanDavis]]
* 03:54 [[etherpad:p/kubecon]] was archived as [[etherpadbackup:kubecon]] by [[User:BryanDavis|BryanDavis]]
* 03:54 [[etherpad:p/kubeadm]] was archived as [[etherpadbackup:kubeadm]] by [[User:BryanDavis|BryanDavis]]
* 03:53 [[etherpad:p/kill-gridengine-slowly]] was archived as [[etherpadbackup:kill-gridengine-slowly]] by [[User:BryanDavis|BryanDavis]]
* 03:53 [[etherpad:p/keystoneldap]] was archived as [[etherpadbackup:keystoneldap]] by [[User:BryanDavis|BryanDavis]]
* 03:52 [[etherpad:p/keystonebootstrap]] was archived as [[etherpadbackup:keystonebootstrap]] by [[User:BryanDavis|BryanDavis]]
* 03:52 [[etherpad:p/kafkaswift]] was archived as [[etherpadbackup:kafkaswift]] by [[User:BryanDavis|BryanDavis]]
* 03:51 [[etherpad:p/kZaM0swM4c]] was archived as [[etherpadbackup:kZaM0swM4c]] by [[User:BryanDavis|BryanDavis]]
* 03:51 [[etherpad:p/kAh7UZSoyI]] was archived as [[etherpadbackup:kAh7UZSoyI]] by [[User:BryanDavis|BryanDavis]]
* 03:50 [[etherpad:p/kAKhhwCsJ7]] was archived as [[etherpadbackup:kAKhhwCsJ7]] by [[User:BryanDavis|BryanDavis]]
* 03:49 [[etherpad:p/k8s-status-wishlist]] was archived as [[etherpadbackup:k8s-status-wishlist]] by [[User:BryanDavis|BryanDavis]]
* 03:49 [[etherpad:p/jupyter-at-wikimedia]] was archived as [[etherpadbackup:jupyter-at-wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 03:48 [[etherpad:p/junk]] was archived as [[etherpadbackup:junk]] by [[User:BryanDavis|BryanDavis]]
* 03:48 [[etherpad:p/joes-random-notes]] was archived as [[etherpadbackup:joes-random-notes]] by [[User:BryanDavis|BryanDavis]]
* 03:47 [[etherpad:p/jmxtrans-packaging-improvements]] was archived as [[etherpadbackup:jmxtrans-packaging-improvements]] by [[User:BryanDavis|BryanDavis]]
* 03:47 [[etherpad:p/jjb]] was archived as [[etherpadbackup:jjb]] by [[User:BryanDavis|BryanDavis]]
* 03:46 [[etherpad:p/jenkins-upgrading]] was archived as [[etherpadbackup:jenkins-upgrading]] by [[User:BryanDavis|BryanDavis]]
* 03:46 [[etherpad:p/jenkins-job-builder]] was archived as [[etherpadbackup:jenkins-job-builder]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/javascript-scoping]] was archived as [[etherpadbackup:javascript-scoping]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/jagori]] was archived as [[etherpadbackup:jagori]] by [[User:BryanDavis|BryanDavis]]
* 03:44 [[etherpad:p/jQUprade]] was archived as [[etherpadbackup:jQUprade]] by [[User:BryanDavis|BryanDavis]]
* 03:44 [[etherpad:p/j9qaVR_wF9bIWn6BWuIr]] was archived as [[etherpadbackup:j9qaVR_wF9bIWn6BWuIr]] by [[User:BryanDavis|BryanDavis]]
* 03:43 [[etherpad:p/j4Obl63I8y]] was archived as [[etherpadbackup:j4Obl63I8y]] by [[User:BryanDavis|BryanDavis]]
* 03:43 [[etherpad:p/iterationname]] was archived as [[etherpadbackup:iterationname]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/is]] was archived as [[etherpadbackup:is]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/ipv6]] was archived as [[etherpadbackup:ipv6]] by [[User:BryanDavis|BryanDavis]]
* 03:41 [[etherpad:p/iosreleasing]] was archived as [[etherpadbackup:iosreleasing]] by [[User:BryanDavis|BryanDavis]]
* 03:41 [[etherpad:p/iosrelease]] was archived as [[etherpadbackup:iosrelease]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/ios_app_update_release_notes]] was archived as [[etherpadbackup:ios_app_update_release_notes]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/ios-5.0]] was archived as [[etherpadbackup:ios-5.0]] by [[User:BryanDavis|BryanDavis]]
* 03:39 [[etherpad:p/ios-3-1beta1-release-notes]] was archived as [[etherpadbackup:ios-3-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 03:39 [[etherpad:p/intro_to_revscores]] was archived as [[etherpadbackup:intro_to_revscores]] by [[User:BryanDavis|BryanDavis]]
* 03:38 [[etherpad:p/internal-graph-split-lvs]] was archived as [[etherpadbackup:internal-graph-split-lvs]] by [[User:BryanDavis|BryanDavis]]
* 03:38 [[etherpad:p/intercontinental-debugging-may-07-2012]] was archived as [[etherpadbackup:intercontinental-debugging-may-07-2012]] by [[User:BryanDavis|BryanDavis]]
* 03:37 [[etherpad:p/incident-20190509-codfwpuppetmasterdown]] was archived as [[etherpadbackup:incident-20190509-codfwpuppetmasterdown]] by [[User:BryanDavis|BryanDavis]]
* 03:37 [[etherpad:p/in_article_search_query]] was archived as [[etherpadbackup:in_article_search_query]] by [[User:BryanDavis|BryanDavis]]
* 03:36 [[etherpad:p/import_wcqs]] was archived as [[etherpadbackup:import_wcqs]] by [[User:BryanDavis|BryanDavis]]
* 03:36 [[etherpad:p/ihatesasl]] was archived as [[etherpadbackup:ihatesasl]] by [[User:BryanDavis|BryanDavis]]
* 03:35 [[etherpad:p/iOS_available_locale_ids]] was archived as [[etherpadbackup:iOS_available_locale_ids]] by [[User:BryanDavis|BryanDavis]]
* 03:34 [[etherpad:p/iOSDataMerge]] was archived as [[etherpadbackup:iOSDataMerge]] by [[User:BryanDavis|BryanDavis]]
* 03:34 [[etherpad:p/iOSAppi18nIssues]] was archived as [[etherpadbackup:iOSAppi18nIssues]] by [[User:BryanDavis|BryanDavis]]
* 03:33 [[etherpad:p/iOSAppDescription]] was archived as [[etherpadbackup:iOSAppDescription]] by [[User:BryanDavis|BryanDavis]]
* 03:33 [[etherpad:p/iOS-ChangeLog]] was archived as [[etherpadbackup:iOS-ChangeLog]] by [[User:BryanDavis|BryanDavis]]
* 03:32 [[etherpad:p/iOS-7-issues]] was archived as [[etherpadbackup:iOS-7-issues]] by [[User:BryanDavis|BryanDavis]]
* 03:32 [[etherpad:p/i8XjsnBxxh]] was archived as [[etherpadbackup:i8XjsnBxxh]] by [[User:BryanDavis|BryanDavis]]
* 03:31 [[etherpad:p/i18n-rfc-2013-11]] was archived as [[etherpadbackup:i18n-rfc-2013-11]] by [[User:BryanDavis|BryanDavis]]
* 03:31 [[etherpad:p/huggle_section]] was archived as [[etherpadbackup:huggle_section]] by [[User:BryanDavis|BryanDavis]]
* 03:30 [[etherpad:p/horizonsso]] was archived as [[etherpadbackup:horizonsso]] by [[User:BryanDavis|BryanDavis]]
* 03:30 [[etherpad:p/hi6ZshFUi2]] was archived as [[etherpadbackup:hi6ZshFUi2]] by [[User:BryanDavis|BryanDavis]]
* 03:29 [[etherpad:p/hhvm]] was archived as [[etherpadbackup:hhvm]] by [[User:BryanDavis|BryanDavis]]
* 03:29 [[etherpad:p/helm-toollabs]] was archived as [[etherpadbackup:helm-toollabs]] by [[User:BryanDavis|BryanDavis]]
* 03:28 [[etherpad:p/helm-chart-development]] was archived as [[etherpadbackup:helm-chart-development]] by [[User:BryanDavis|BryanDavis]]
* 03:28 [[etherpad:p/handoff-wdqs-T388134]] was archived as [[etherpadbackup:handoff-wdqs-T388134]] by [[User:BryanDavis|BryanDavis]]
* 03:27 [[etherpad:p/hackathon-showcase-before-the-catastrophe]] was archived as [[etherpadbackup:hackathon-showcase-before-the-catastrophe]] by [[User:BryanDavis|BryanDavis]]
* 03:27 [[etherpad:p/hackathon-rhinosf1-barriers]] was archived as [[etherpadbackup:hackathon-rhinosf1-barriers]] by [[User:BryanDavis|BryanDavis]]
* 03:26 [[etherpad:p/gridengine-uses]] was archived as [[etherpadbackup:gridengine-uses]] by [[User:BryanDavis|BryanDavis]]
* 03:26 [[etherpad:p/grid-backup]] was archived as [[etherpadbackup:grid-backup]] by [[User:BryanDavis|BryanDavis]]
* 03:25 [[etherpad:p/graphitetodo]] was archived as [[etherpadbackup:graphitetodo]] by [[User:BryanDavis|BryanDavis]]
* 03:25 [[etherpad:p/grantreview]] was archived as [[etherpadbackup:grantreview]] by [[User:BryanDavis|BryanDavis]]
* 03:24 [[etherpad:p/gitlab-sync]] was archived as [[etherpadbackup:gitlab-sync]] by [[User:BryanDavis|BryanDavis]]
* 03:24 [[etherpad:p/git]] was archived as [[etherpadbackup:git]] by [[User:BryanDavis|BryanDavis]]
* 03:23 [[etherpad:p/gerrit-4841]] was archived as [[etherpadbackup:gerrit-4841]] by [[User:BryanDavis|BryanDavis]]
* 03:23 [[etherpad:p/gdash-labels-clarification]] was archived as [[etherpadbackup:gdash-labels-clarification]] by [[User:BryanDavis|BryanDavis]]
* 03:22 [[etherpad:p/gcimail]] was archived as [[etherpadbackup:gcimail]] by [[User:BryanDavis|BryanDavis]]
* 03:22 [[etherpad:p/gannum]] was archived as [[etherpadbackup:gannum]] by [[User:BryanDavis|BryanDavis]]
* 03:21 [[etherpad:p/g5an_WR4fa02lVjmHQse]] was archived as [[etherpadbackup:g5an_WR4fa02lVjmHQse]] by [[User:BryanDavis|BryanDavis]]
* 03:21 [[etherpad:p/g505410announce]] was archived as [[etherpadbackup:g505410announce]] by [[User:BryanDavis|BryanDavis]]
* 03:20 [[etherpad:p/g4-flavors]] was archived as [[etherpadbackup:g4-flavors]] by [[User:BryanDavis|BryanDavis]]
* 03:19 [[etherpad:p/future-of-language-converter]] was archived as [[etherpadbackup:future-of-language-converter]] by [[User:BryanDavis|BryanDavis]]
* 03:19 [[etherpad:p/fullstack_leaks]] was archived as [[etherpadbackup:fullstack_leaks]] by [[User:BryanDavis|BryanDavis]]
* 03:18 [[etherpad:p/fullstack-flapping]] was archived as [[etherpadbackup:fullstack-flapping]] by [[User:BryanDavis|BryanDavis]]
* 03:18 [[etherpad:p/fr-tech_chores_2023_fish_HEAD^]] was archived as [[etherpadbackup:fr-tech_chores_2023_fish_HEAD^]] by [[User:BryanDavis|BryanDavis]]
* 03:17 [[etherpad:p/fp-Hrk0fve_GXMeMZFJk]] was archived as [[etherpadbackup:fp-Hrk0fve_GXMeMZFJk]] by [[User:BryanDavis|BryanDavis]]
* 03:17 [[etherpad:p/floating-ip-aliaser]] was archived as [[etherpadbackup:floating-ip-aliaser]] by [[User:BryanDavis|BryanDavis]]
* 03:16 [[etherpad:p/flagellating-funnel]] was archived as [[etherpadbackup:flagellating-funnel]] by [[User:BryanDavis|BryanDavis]]
* 03:16 [[etherpad:p/extdist]] was archived as [[etherpadbackup:extdist]] by [[User:BryanDavis|BryanDavis]]
* 03:15 [[etherpad:p/existensions]] was archived as [[etherpadbackup:existensions]] by [[User:BryanDavis|BryanDavis]]
* 03:15 [[etherpad:p/eventlogging_stag]] was archived as [[etherpadbackup:eventlogging_stag]] by [[User:BryanDavis|BryanDavis]]
* 03:14 [[etherpad:p/eventbus-statsd]] was archived as [[etherpadbackup:eventbus-statsd]] by [[User:BryanDavis|BryanDavis]]
* 03:14 [[etherpad:p/essex-upgrade]] was archived as [[etherpadbackup:essex-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 03:13 [[etherpad:p/esams-followup]] was archived as [[etherpadbackup:esams-followup]] by [[User:BryanDavis|BryanDavis]]
* 03:13 [[etherpad:p/eqiad1-upgrade-pike]] was archived as [[etherpadbackup:eqiad1-upgrade-pike]] by [[User:BryanDavis|BryanDavis]]
* 03:12 [[etherpad:p/eqiad1-neutron-bootstrap]] was archived as [[etherpadbackup:eqiad1-neutron-bootstrap]] by [[User:BryanDavis|BryanDavis]]
* 03:12 [[etherpad:p/eqiad1-keystone-bootstrap]] was archived as [[etherpadbackup:eqiad1-keystone-bootstrap]] by [[User:BryanDavis|BryanDavis]]
* 03:11 [[etherpad:p/eqiad1]] was archived as [[etherpadbackup:eqiad1]] by [[User:BryanDavis|BryanDavis]]
* 03:11 [[etherpad:p/enwikiSOPADBUpgrades]] was archived as [[etherpadbackup:enwikiSOPADBUpgrades]] by [[User:BryanDavis|BryanDavis]]
* 03:10 [[etherpad:p/engprod-irc]] was archived as [[etherpadbackup:engprod-irc]] by [[User:BryanDavis|BryanDavis]]
* 03:10 [[etherpad:p/empty_vps_projects]] was archived as [[etherpadbackup:empty_vps_projects]] by [[User:BryanDavis|BryanDavis]]
* 03:09 [[etherpad:p/embeddings_and_topic_models]] was archived as [[etherpadbackup:embeddings_and_topic_models]] by [[User:BryanDavis|BryanDavis]]
* 03:09 [[etherpad:p/email-tools]] was archived as [[etherpadbackup:email-tools]] by [[User:BryanDavis|BryanDavis]]
* 03:08 [[etherpad:p/elukey-netflow]] was archived as [[etherpadbackup:elukey-netflow]] by [[User:BryanDavis|BryanDavis]]
* 03:07 [[etherpad:p/elastic-single-node-testing]] was archived as [[etherpadbackup:elastic-single-node-testing]] by [[User:BryanDavis|BryanDavis]]
* 03:07 [[etherpad:p/elastic-2-opensearch-T388610]] was archived as [[etherpadbackup:elastic-2-opensearch-T388610]] by [[User:BryanDavis|BryanDavis]]
* 03:06 [[etherpad:p/el_utils]] was archived as [[etherpadbackup:el_utils]] by [[User:BryanDavis|BryanDavis]]
* 03:06 [[etherpad:p/eiwGl_RxtBIFSyuxv5Yr]] was archived as [[etherpadbackup:eiwGl_RxtBIFSyuxv5Yr]] by [[User:BryanDavis|BryanDavis]]
* 03:05 [[etherpad:p/effiee]] was archived as [[etherpadbackup:effiee]] by [[User:BryanDavis|BryanDavis]]
* 03:05 [[etherpad:p/editing_logging]] was archived as [[etherpadbackup:editing_logging]] by [[User:BryanDavis|BryanDavis]]
* 03:04 [[etherpad:p/edit_types_event_schema]] was archived as [[etherpadbackup:edit_types_event_schema]] by [[User:BryanDavis|BryanDavis]]
* 03:04 [[etherpad:p/edit_history_vetting]] was archived as [[etherpadbackup:edit_history_vetting]] by [[User:BryanDavis|BryanDavis]]
* 03:03 [[etherpad:p/e1]] was archived as [[etherpadbackup:e1]] by [[User:BryanDavis|BryanDavis]]
* 03:03 [[etherpad:p/e]] was archived as [[etherpadbackup:e]] by [[User:BryanDavis|BryanDavis]]
* 03:02 [[etherpad:p/doc-from-git]] was archived as [[etherpadbackup:doc-from-git]] by [[User:BryanDavis|BryanDavis]]
* 03:02 [[etherpad:p/dnsthings]] was archived as [[etherpadbackup:dnsthings]] by [[User:BryanDavis|BryanDavis]]
* 03:01 [[etherpad:p/dns_cleanup]] was archived as [[etherpadbackup:dns_cleanup]] by [[User:BryanDavis|BryanDavis]]
* 03:01 [[etherpad:p/dmitry-onboarding]] was archived as [[etherpadbackup:dmitry-onboarding]] by [[User:BryanDavis|BryanDavis]]
* 03:00 [[etherpad:p/disk_aio_setting]] was archived as [[etherpadbackup:disk_aio_setting]] by [[User:BryanDavis|BryanDavis]]
* 03:00 [[etherpad:p/diff_test]] was archived as [[etherpadbackup:diff_test]] by [[User:BryanDavis|BryanDavis]]
* 02:59 [[etherpad:p/diamond-deployment]] was archived as [[etherpadbackup:diamond-deployment]] by [[User:BryanDavis|BryanDavis]]
* 02:59 [[etherpad:p/dhcp-option82-sunset]] was archived as [[etherpadbackup:dhcp-option82-sunset]] by [[User:BryanDavis|BryanDavis]]
* 02:58 [[etherpad:p/devsummit18-thirdpartymediawiki]] was archived as [[etherpadbackup:devsummit18-thirdpartymediawiki]] by [[User:BryanDavis|BryanDavis]]
* 02:58 [[etherpad:p/devsummit17-support-global-preferences]] was archived as [[etherpadbackup:devsummit17-support-global-preferences]] by [[User:BryanDavis|BryanDavis]]
* 02:57 [[etherpad:p/devsummit17-scaling-database-schema]] was archived as [[etherpadbackup:devsummit17-scaling-database-schema]] by [[User:BryanDavis|BryanDavis]]
* 02:57 [[etherpad:p/devsummit17-integrating-mediawiki]] was archived as [[etherpadbackup:devsummit17-integrating-mediawiki]] by [[User:BryanDavis|BryanDavis]]
* 02:56 [[etherpad:p/devsummit17-algorithmic-dangers]] was archived as [[etherpadbackup:devsummit17-algorithmic-dangers]] by [[User:BryanDavis|BryanDavis]]
* 02:56 [[etherpad:p/devsummit17-ai-wishlist]] was archived as [[etherpadbackup:devsummit17-ai-wishlist]] by [[User:BryanDavis|BryanDavis]]
* 02:55 [[etherpad:p/devsummit17-T149624]] was archived as [[etherpadbackup:devsummit17-T149624]] by [[User:BryanDavis|BryanDavis]]
* 02:55 [[etherpad:p/devsummit17-RedesignSpecialSearchPage]] was archived as [[etherpadbackup:devsummit17-RedesignSpecialSearchPage]] by [[User:BryanDavis|BryanDavis]]
* 02:54 [[etherpad:p/devsummit17-ProjectOREO]] was archived as [[etherpadbackup:devsummit17-ProjectOREO]] by [[User:BryanDavis|BryanDavis]]
* 02:53 [[etherpad:p/devsummit17-CodeReview]] was archived as [[etherpadbackup:devsummit17-CodeReview]] by [[User:BryanDavis|BryanDavis]]
* 02:53 [[etherpad:p/devpowpow-2015-02-25]] was archived as [[etherpadbackup:devpowpow-2015-02-25]] by [[User:BryanDavis|BryanDavis]]
* 02:52 [[etherpad:p/design_review_10_22]] was archived as [[etherpadbackup:design_review_10_22]] by [[User:BryanDavis|BryanDavis]]
* 02:52 [[etherpad:p/design-thingies-jan-android]] was archived as [[etherpadbackup:design-thingies-jan-android]] by [[User:BryanDavis|BryanDavis]]
* 02:51 [[etherpad:p/design]] was archived as [[etherpadbackup:design]] by [[User:BryanDavis|BryanDavis]]
* 02:51 [[etherpad:p/deprecate-precise]] was archived as [[etherpadbackup:deprecate-precise]] by [[User:BryanDavis|BryanDavis]]
* 02:50 [[etherpad:p/depool]] was archived as [[etherpadbackup:depool]] by [[User:BryanDavis|BryanDavis]]
* 02:50 [[etherpad:p/deployworkinggroup]] was archived as [[etherpadbackup:deployworkinggroup]] by [[User:BryanDavis|BryanDavis]]
* 02:49 [[etherpad:p/deploytrainingprep]] was archived as [[etherpadbackup:deploytrainingprep]] by [[User:BryanDavis|BryanDavis]]
* 02:49 [[etherpad:p/deploy-20150427-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150427-SWAT-evening]] by [[User:BryanDavis|BryanDavis]]
* 02:48 [[etherpad:p/deploy-20150416-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150416-SWAT-evening]] by [[User:BryanDavis|BryanDavis]]
* 02:48 [[etherpad:p/deletedesedomains]] was archived as [[etherpadbackup:deletedesedomains]] by [[User:BryanDavis|BryanDavis]]
* 02:47 [[etherpad:p/dear-tool-user]] was archived as [[etherpadbackup:dear-tool-user]] by [[User:BryanDavis|BryanDavis]]
* 02:47 [[etherpad:p/dchanPlate2015]] was archived as [[etherpadbackup:dchanPlate2015]] by [[User:BryanDavis|BryanDavis]]
* 02:46 [[etherpad:p/db-renames]] was archived as [[etherpadbackup:db-renames]] by [[User:BryanDavis|BryanDavis]]
* 02:46 [[etherpad:p/datalossjuly2]] was archived as [[etherpadbackup:datalossjuly2]] by [[User:BryanDavis|BryanDavis]]
* 02:45 [[etherpad:p/cx-presentation]] was archived as [[etherpadbackup:cx-presentation]] by [[User:BryanDavis|BryanDavis]]
* 02:45 [[etherpad:p/cx-markup-alignment-uppercase]] was archived as [[etherpadbackup:cx-markup-alignment-uppercase]] by [[User:BryanDavis|BryanDavis]]
* 02:44 [[etherpad:p/cx-markup-alignment]] was archived as [[etherpadbackup:cx-markup-alignment]] by [[User:BryanDavis|BryanDavis]]
* 02:44 [[etherpad:p/cx-invite]] was archived as [[etherpadbackup:cx-invite]] by [[User:BryanDavis|BryanDavis]]
* 02:43 [[etherpad:p/cx-hack]] was archived as [[etherpadbackup:cx-hack]] by [[User:BryanDavis|BryanDavis]]
* 02:43 [[etherpad:p/cx-composer]] was archived as [[etherpadbackup:cx-composer]] by [[User:BryanDavis|BryanDavis]]
* 02:42 [[etherpad:p/csKD3No9oL]] was archived as [[etherpadbackup:csKD3No9oL]] by [[User:BryanDavis|BryanDavis]]
* 02:42 [[etherpad:p/cronerror]] was archived as [[etherpadbackup:cronerror]] by [[User:BryanDavis|BryanDavis]]
* 02:41 [[etherpad:p/cr2-codfw_fpc0]] was archived as [[etherpadbackup:cr2-codfw_fpc0]] by [[User:BryanDavis|BryanDavis]]
* 02:41 [[etherpad:p/conftool-T379329]] was archived as [[etherpadbackup:conftool-T379329]] by [[User:BryanDavis|BryanDavis]]
* 02:40 [[etherpad:p/completion-ab-sanity-check]] was archived as [[etherpadbackup:completion-ab-sanity-check]] by [[User:BryanDavis|BryanDavis]]
* 02:40 [[etherpad:p/compat-network]] was archived as [[etherpadbackup:compat-network]] by [[User:BryanDavis|BryanDavis]]
* 02:39 [[etherpad:p/commons-ios-fixes]] was archived as [[etherpadbackup:commons-ios-fixes]] by [[User:BryanDavis|BryanDavis]]
* 02:38 [[etherpad:p/commons-cross-wiki-uploads]] was archived as [[etherpadbackup:commons-cross-wiki-uploads]] by [[User:BryanDavis|BryanDavis]]
* 02:38 [[etherpad:p/commons-app-post-draft]] was archived as [[etherpadbackup:commons-app-post-draft]] by [[User:BryanDavis|BryanDavis]]
* 02:37 [[etherpad:p/collation]] was archived as [[etherpadbackup:collation]] by [[User:BryanDavis|BryanDavis]]
* 02:37 [[etherpad:p/codeacademy-diwanship]] was archived as [[etherpadbackup:codeacademy-diwanship]] by [[User:BryanDavis|BryanDavis]]
* 02:36 [[etherpad:p/cna4GMpRlJCm65THgk-l]] was archived as [[etherpadbackup:cna4GMpRlJCm65THgk-l]] by [[User:BryanDavis|BryanDavis]]
* 02:36 [[etherpad:p/cloudvpskernels]] was archived as [[etherpadbackup:cloudvpskernels]] by [[User:BryanDavis|BryanDavis]]
* 02:35 [[etherpad:p/cloudvps-no-agent-forwarding]] was archived as [[etherpadbackup:cloudvps-no-agent-forwarding]] by [[User:BryanDavis|BryanDavis]]
* 02:35 [[etherpad:p/cloudvirt1018-20190213]] was archived as [[etherpadbackup:cloudvirt1018-20190213]] by [[User:BryanDavis|BryanDavis]]
* 02:34 [[etherpad:p/cloudservices1003-upgrade]] was archived as [[etherpadbackup:cloudservices1003-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 02:34 [[etherpad:p/cloudreboots]] was archived as [[etherpadbackup:cloudreboots]] by [[User:BryanDavis|BryanDavis]]
* 02:33 [[etherpad:p/cloudprojectrequeststweak]] was archived as [[etherpadbackup:cloudprojectrequeststweak]] by [[User:BryanDavis|BryanDavis]]
* 02:33 [[etherpad:p/cloudnative]] was archived as [[etherpadbackup:cloudnative]] by [[User:BryanDavis|BryanDavis]]
* 02:32 [[etherpad:p/cloudgw-migration]] was archived as [[etherpadbackup:cloudgw-migration]] by [[User:BryanDavis|BryanDavis]]
* 02:32 [[etherpad:p/cloudfontcdn]] was archived as [[etherpadbackup:cloudfontcdn]] by [[User:BryanDavis|BryanDavis]]
* 02:31 [[etherpad:p/cloudelastic-mystery-red]] was archived as [[etherpadbackup:cloudelastic-mystery-red]] by [[User:BryanDavis|BryanDavis]]
* 02:31 [[etherpad:p/cloudelastic-ipip]] was archived as [[etherpadbackup:cloudelastic-ipip]] by [[User:BryanDavis|BryanDavis]]
* 02:30 [[etherpad:p/cloudelastic-2-opensearch]] was archived as [[etherpadbackup:cloudelastic-2-opensearch]] by [[User:BryanDavis|BryanDavis]]
* 02:30 [[etherpad:p/cloud_purge_2018_ping]] was archived as [[etherpadbackup:cloud_purge_2018_ping]] by [[User:BryanDavis|BryanDavis]]
* 02:29 [[etherpad:p/cloud_announcement_20180910]] was archived as [[etherpadbackup:cloud_announcement_20180910]] by [[User:BryanDavis|BryanDavis]]
* 02:29 [[etherpad:p/cloud-vps-purge-deletions]] was archived as [[etherpadbackup:cloud-vps-purge-deletions]] by [[User:BryanDavis|BryanDavis]]
* 02:28 [[etherpad:p/citoid-TD-fun]] was archived as [[etherpadbackup:citoid-TD-fun]] by [[User:BryanDavis|BryanDavis]]
* 02:28 [[etherpad:p/cirrus_cross_project_search_debug_settings]] was archived as [[etherpadbackup:cirrus_cross_project_search_debug_settings]] by [[User:BryanDavis|BryanDavis]]
* 02:27 [[etherpad:p/ci-apt-docker-why]] was archived as [[etherpadbackup:ci-apt-docker-why]] by [[User:BryanDavis|BryanDavis]]
* 02:27 [[etherpad:p/chrismcmahon]] was archived as [[etherpadbackup:chrismcmahon]] by [[User:BryanDavis|BryanDavis]]
* 02:26 [[etherpad:p/changeuid]] was archived as [[etherpadbackup:changeuid]] by [[User:BryanDavis|BryanDavis]]
* 02:26 [[etherpad:p/changes]] was archived as [[etherpadbackup:changes]] by [[User:BryanDavis|BryanDavis]]
* 02:25 [[etherpad:p/ceph-poc-to-prod]] was archived as [[etherpadbackup:ceph-poc-to-prod]] by [[User:BryanDavis|BryanDavis]]
* 02:25 [[etherpad:p/categories-lag]] was archived as [[etherpadbackup:categories-lag]] by [[User:BryanDavis|BryanDavis]]
* 02:24 [[etherpad:p/calendar-minimums]] was archived as [[etherpadbackup:calendar-minimums]] by [[User:BryanDavis|BryanDavis]]
* 02:24 [[etherpad:p/busterreminder]] was archived as [[etherpadbackup:busterreminder]] by [[User:BryanDavis|BryanDavis]]
* 02:23 [[etherpad:p/bug35939]] was archived as [[etherpadbackup:bug35939]] by [[User:BryanDavis|BryanDavis]]
* 02:22 [[etherpad:p/brokenpms]] was archived as [[etherpadbackup:brokenpms]] by [[User:BryanDavis|BryanDavis]]
* 02:22 [[etherpad:p/bloated-website-code-drains-yo]] was archived as [[etherpadbackup:bloated-website-code-drains-yo]] by [[User:BryanDavis|BryanDavis]]
* 02:21 [[etherpad:p/bios-be-gone]] was archived as [[etherpadbackup:bios-be-gone]] by [[User:BryanDavis|BryanDavis]]
* 02:21 [[etherpad:p/bigbrother-almost-no-more]] was archived as [[etherpadbackup:bigbrother-almost-no-more]] by [[User:BryanDavis|BryanDavis]]
* 02:20 [[etherpad:p/beta-new-ext]] was archived as [[etherpadbackup:beta-new-ext]] by [[User:BryanDavis|BryanDavis]]
* 02:20 [[etherpad:p/bernd-onboarding]] was archived as [[etherpadbackup:bernd-onboarding]] by [[User:BryanDavis|BryanDavis]]
* 02:19 [[etherpad:p/beginner-hackathons-thoughts]] was archived as [[etherpadbackup:beginner-hackathons-thoughts]] by [[User:BryanDavis|BryanDavis]]
* 02:19 [[etherpad:p/bash]] was archived as [[etherpadbackup:bash]] by [[User:BryanDavis|BryanDavis]]
* 02:18 [[etherpad:p/bangarang]] was archived as [[etherpadbackup:bangarang]] by [[User:BryanDavis|BryanDavis]]
* 02:18 [[etherpad:p/band_names_web]] was archived as [[etherpadbackup:band_names_web]] by [[User:BryanDavis|BryanDavis]]
* 02:17 [[etherpad:p/avro]] was archived as [[etherpadbackup:avro]] by [[User:BryanDavis|BryanDavis]]
* 02:17 [[etherpad:p/aqs-standup]] was archived as [[etherpadbackup:aqs-standup]] by [[User:BryanDavis|BryanDavis]]
* 02:16 [[etherpad:p/app_q3_planning]] was archived as [[etherpadbackup:app_q3_planning]] by [[User:BryanDavis|BryanDavis]]
* 02:16 [[etherpad:p/app_q3_health_check]] was archived as [[etherpadbackup:app_q3_health_check]] by [[User:BryanDavis|BryanDavis]]
* 02:15 [[etherpad:p/app_process_improvements]] was archived as [[etherpadbackup:app_process_improvements]] by [[User:BryanDavis|BryanDavis]]
* 02:15 [[etherpad:p/appQ4planning]] was archived as [[etherpadbackup:appQ4planning]] by [[User:BryanDavis|BryanDavis]]
* 02:14 [[etherpad:p/app-strategy-discussion]] was archived as [[etherpadbackup:app-strategy-discussion]] by [[User:BryanDavis|BryanDavis]]
* 02:14 [[etherpad:p/app-server-upgrade]] was archived as [[etherpadbackup:app-server-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 02:13 [[etherpad:p/app-1-1beta4-3-1beta3-release-notes]] was archived as [[etherpadbackup:app-1-1beta4-3-1beta3-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:13 [[etherpad:p/apiv2]] was archived as [[etherpadbackup:apiv2]] by [[User:BryanDavis|BryanDavis]]
* 02:12 [[etherpad:p/api.php]] was archived as [[etherpadbackup:api.php]] by [[User:BryanDavis|BryanDavis]]
* 02:12 [[etherpad:p/androidappfeedback]] was archived as [[etherpadbackup:androidappfeedback]] by [[User:BryanDavis|BryanDavis]]
* 02:11 [[etherpad:p/android-ime]] was archived as [[etherpadbackup:android-ime]] by [[User:BryanDavis|BryanDavis]]
* 02:11 [[etherpad:p/android-commons-app-1-0]] was archived as [[etherpadbackup:android-commons-app-1-0]] by [[User:BryanDavis|BryanDavis]]
* 02:10 [[etherpad:p/android-1-1rc1-release-notes]] was archived as [[etherpadbackup:android-1-1rc1-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:10 [[etherpad:p/android-1-1beta1-release-notes]] was archived as [[etherpadbackup:android-1-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:09 [[etherpad:p/android-1-1-iOS-3-1-third-beta-notes]] was archived as [[etherpadbackup:android-1-1-iOS-3-1-third-beta-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:08 [[etherpad:p/android-1-1-beta2-release-notes]] was archived as [[etherpadbackup:android-1-1-beta2-release-notes]] by [[User:BryanDavis|BryanDavis]]
* 02:08 [[etherpad:p/analytics163]] was archived as [[etherpadbackup:analytics163]] by [[User:BryanDavis|BryanDavis]]
* 02:07 [[etherpad:p/analytics1003-reinstall]] was archived as [[etherpadbackup:analytics1003-reinstall]] by [[User:BryanDavis|BryanDavis]]
* 02:07 [[etherpad:p/analytics-todo]] was archived as [[etherpadbackup:analytics-todo]] by [[User:BryanDavis|BryanDavis]]
* 02:06 [[etherpad:p/analytics-tech-debt]] was archived as [[etherpadbackup:analytics-tech-debt]] by [[User:BryanDavis|BryanDavis]]
* 02:06 [[etherpad:p/analytics-staff-meeting]] was archived as [[etherpadbackup:analytics-staff-meeting]] by [[User:BryanDavis|BryanDavis]]
* 02:05 [[etherpad:p/analytics-row-d-maintenance]] was archived as [[etherpadbackup:analytics-row-d-maintenance]] by [[User:BryanDavis|BryanDavis]]
* 02:05 [[etherpad:p/analytics-retrospective]] was archived as [[etherpadbackup:analytics-retrospective]] by [[User:BryanDavis|BryanDavis]]
* 02:04 [[etherpad:p/analytics-ops-druid]] was archived as [[etherpadbackup:analytics-ops-druid]] by [[User:BryanDavis|BryanDavis]]
* 02:04 [[etherpad:p/analytics-ops]] was archived as [[etherpadbackup:analytics-ops]] by [[User:BryanDavis|BryanDavis]]
* 02:03 [[etherpad:p/analytics-oozie-13102016]] was archived as [[etherpadbackup:analytics-oozie-13102016]] by [[User:BryanDavis|BryanDavis]]
* 02:03 [[etherpad:p/analytics-naming]] was archived as [[etherpadbackup:analytics-naming]] by [[User:BryanDavis|BryanDavis]]
* 02:02 [[etherpad:p/analytics-mwds-2015]] was archived as [[etherpadbackup:analytics-mwds-2015]] by [[User:BryanDavis|BryanDavis]]
* 02:02 [[etherpad:p/analytics-ingestion]] was archived as [[etherpadbackup:analytics-ingestion]] by [[User:BryanDavis|BryanDavis]]
* 02:01 [[etherpad:p/analytics-hive-udf]] was archived as [[etherpadbackup:analytics-hive-udf]] by [[User:BryanDavis|BryanDavis]]
* 02:01 [[etherpad:p/analytics-gobblin-mess]] was archived as [[etherpadbackup:analytics-gobblin-mess]] by [[User:BryanDavis|BryanDavis]]
* 02:00 [[etherpad:p/analytics-goals]] was archived as [[etherpadbackup:analytics-goals]] by [[User:BryanDavis|BryanDavis]]
* 02:00 [[etherpad:p/analytics-eventlogging-postmortem-2014-11]] was archived as [[etherpadbackup:analytics-eventlogging-postmortem-2014-11]] by [[User:BryanDavis|BryanDavis]]
* 01:59 [[etherpad:p/analytics-email-drafts]] was archived as [[etherpadbackup:analytics-email-drafts]] by [[User:BryanDavis|BryanDavis]]
* 01:59 [[etherpad:p/analytics-druid-upgrade]] was archived as [[etherpadbackup:analytics-druid-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:58 [[etherpad:p/analytics-druid-migration]] was archived as [[etherpadbackup:analytics-druid-migration]] by [[User:BryanDavis|BryanDavis]]
* 01:58 [[etherpad:p/analytics-deploy-aqs]] was archived as [[etherpadbackup:analytics-deploy-aqs]] by [[User:BryanDavis|BryanDavis]]
* 01:57 [[etherpad:p/analytics-bikeshedding]] was archived as [[etherpadbackup:analytics-bikeshedding]] by [[User:BryanDavis|BryanDavis]]
* 01:57 [[etherpad:p/analytics-bikeshed]] was archived as [[etherpadbackup:analytics-bikeshed]] by [[User:BryanDavis|BryanDavis]]
* 01:56 [[etherpad:p/analytics-analytics1003-jessie-upgrade]] was archived as [[etherpadbackup:analytics-analytics1003-jessie-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:56 [[etherpad:p/analytics-VE]] was archived as [[etherpadbackup:analytics-VE]] by [[User:BryanDavis|BryanDavis]]
* 01:55 [[etherpad:p/analytics-73331]] was archived as [[etherpadbackup:analytics-73331]] by [[User:BryanDavis|BryanDavis]]
* 01:55 [[etherpad:p/analytics-72745]] was archived as [[etherpadbackup:analytics-72745]] by [[User:BryanDavis|BryanDavis]]
* 01:54 [[etherpad:p/analytics-72740]] was archived as [[etherpadbackup:analytics-72740]] by [[User:BryanDavis|BryanDavis]]
* 01:54 [[etherpad:p/analytics-72739]] was archived as [[etherpadbackup:analytics-72739]] by [[User:BryanDavis|BryanDavis]]
* 01:53 [[etherpad:p/analytics-72738]] was archived as [[etherpadbackup:analytics-72738]] by [[User:BryanDavis|BryanDavis]]
* 01:52 [[etherpad:p/analytics-72737]] was archived as [[etherpadbackup:analytics-72737]] by [[User:BryanDavis|BryanDavis]]
* 01:52 [[etherpad:p/analytics-72736]] was archived as [[etherpadbackup:analytics-72736]] by [[User:BryanDavis|BryanDavis]]
* 01:51 [[etherpad:p/analytics-72735]] was archived as [[etherpadbackup:analytics-72735]] by [[User:BryanDavis|BryanDavis]]
* 01:51 [[etherpad:p/analytics-69254]] was archived as [[etherpadbackup:analytics-69254]] by [[User:BryanDavis|BryanDavis]]
* 01:50 [[etherpad:p/alpha-signature-change]] was archived as [[etherpadbackup:alpha-signature-change]] by [[User:BryanDavis|BryanDavis]]
* 01:50 [[etherpad:p/ai_vision_wikimedia]] was archived as [[etherpadbackup:ai_vision_wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 01:49 [[etherpad:p/aharoni-20160420]] was archived as [[etherpadbackup:aharoni-20160420]] by [[User:BryanDavis|BryanDavis]]
* 01:49 [[etherpad:p/aftereffects]] was archived as [[etherpadbackup:aftereffects]] by [[User:BryanDavis|BryanDavis]]
* 01:48 [[etherpad:p/admin_accounts_cleanup]] was archived as [[etherpadbackup:admin_accounts_cleanup]] by [[User:BryanDavis|BryanDavis]]
* 01:48 [[etherpad:p/admin]] was archived as [[etherpadbackup:admin]] by [[User:BryanDavis|BryanDavis]]
* 01:47 [[etherpad:p/addwiki]] was archived as [[etherpadbackup:addwiki]] by [[User:BryanDavis|BryanDavis]]
* 01:47 [[etherpad:p/addshore]] was archived as [[etherpadbackup:addshore]] by [[User:BryanDavis|BryanDavis]]
* 01:46 [[etherpad:p/actionbar_checklist]] was archived as [[etherpadbackup:actionbar_checklist]] by [[User:BryanDavis|BryanDavis]]
* 01:46 [[etherpad:p/access_request_wikis]] was archived as [[etherpadbackup:access_request_wikis]] by [[User:BryanDavis|BryanDavis]]
* 01:45 [[etherpad:p/absents]] was archived as [[etherpadbackup:absents]] by [[User:BryanDavis|BryanDavis]]
* 01:45 [[etherpad:p/a4NCijsjQp]] was archived as [[etherpadbackup:a4NCijsjQp]] by [[User:BryanDavis|BryanDavis]]
* 01:44 [[etherpad:p/ZVFiVE3E09]] was archived as [[etherpadbackup:ZVFiVE3E09]] by [[User:BryanDavis|BryanDavis]]
* 01:44 [[etherpad:p/XcC5u59nDdRc_itxINnb]] was archived as [[etherpadbackup:XcC5u59nDdRc_itxINnb]] by [[User:BryanDavis|BryanDavis]]
* 01:43 [[etherpad:p/XH6WkSmARmw8dy7UMiIR]] was archived as [[etherpadbackup:XH6WkSmARmw8dy7UMiIR]] by [[User:BryanDavis|BryanDavis]]
* 01:43 [[etherpad:p/WsWWUiRllf]] was archived as [[etherpadbackup:WsWWUiRllf]] by [[User:BryanDavis|BryanDavis]]
* 01:42 [[etherpad:p/WorT2b8qpR]] was archived as [[etherpadbackup:WorT2b8qpR]] by [[User:BryanDavis|BryanDavis]]
* 01:42 [[etherpad:p/Wj4HcOWoHk]] was archived as [[etherpadbackup:Wj4HcOWoHk]] by [[User:BryanDavis|BryanDavis]]
* 01:41 [[etherpad:p/WikipediaiOSTesting]] was archived as [[etherpadbackup:WikipediaiOSTesting]] by [[User:BryanDavis|BryanDavis]]
* 01:41 [[etherpad:p/WikipediaZeroRetrospective]] was archived as [[etherpadbackup:WikipediaZeroRetrospective]] by [[User:BryanDavis|BryanDavis]]
* 01:40 [[etherpad:p/WikipediaPhoneGapAndroidMeetup]] was archived as [[etherpadbackup:WikipediaPhoneGapAndroidMeetup]] by [[User:BryanDavis|BryanDavis]]
* 01:40 [[etherpad:p/WikipediaMobileiOS3-1Changelog]] was archived as [[etherpadbackup:WikipediaMobileiOS3-1Changelog]] by [[User:BryanDavis|BryanDavis]]
* 01:39 [[etherpad:p/WikipediaMobileAndroidV1-0-1]] was archived as [[etherpadbackup:WikipediaMobileAndroidV1-0-1]] by [[User:BryanDavis|BryanDavis]]
* 01:39 [[etherpad:p/WikipediaMobileAndroidRelease]] was archived as [[etherpadbackup:WikipediaMobileAndroidRelease]] by [[User:BryanDavis|BryanDavis]]
* 01:38 [[etherpad:p/WikipediaMobileAndroid-V1-0-2]] was archived as [[etherpadbackup:WikipediaMobileAndroid-V1-0-2]] by [[User:BryanDavis|BryanDavis]]
* 01:37 [[etherpad:p/WikipediaMobile-3-1-1-scratchpad]] was archived as [[etherpadbackup:WikipediaMobile-3-1-1-scratchpad]] by [[User:BryanDavis|BryanDavis]]
* 01:37 [[etherpad:p/WikipediaLite]] was archived as [[etherpadbackup:WikipediaLite]] by [[User:BryanDavis|BryanDavis]]
* 01:36 [[etherpad:p/Wikipedia-iOS-Rejection-Message]] was archived as [[etherpadbackup:Wikipedia-iOS-Rejection-Message]] by [[User:BryanDavis|BryanDavis]]
* 01:36 [[etherpad:p/Wikipedia-iOS-3-1-1-issues]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1-issues]] by [[User:BryanDavis|BryanDavis]]
* 01:35 [[etherpad:p/Wikipedia-iOS-3-1-1]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1]] by [[User:BryanDavis|BryanDavis]]
* 01:35 [[etherpad:p/Wikipedia-iOS]] was archived as [[etherpadbackup:Wikipedia-iOS]] by [[User:BryanDavis|BryanDavis]]
* 01:34 [[etherpad:p/WikipdiaMobileRC4]] was archived as [[etherpadbackup:WikipdiaMobileRC4]] by [[User:BryanDavis|BryanDavis]]
* 01:34 [[etherpad:p/WikimediaTelepresence]] was archived as [[etherpadbackup:WikimediaTelepresence]] by [[User:BryanDavis|BryanDavis]]
* 01:33 [[etherpad:p/Wikimania_Hackathon_2025_-_Opening_Ceremony]] was archived as [[etherpadbackup:Wikimania_Hackathon_2025_-_Opening_Ceremony]] by [[User:BryanDavis|BryanDavis]]
* 01:33 [[etherpad:p/Wikimania_2023_Hackathon]] was archived as [[etherpadbackup:Wikimania_2023_Hackathon]] by [[User:BryanDavis|BryanDavis]]
* 01:32 [[etherpad:p/Wikigrok_Q4]] was archived as [[etherpadbackup:Wikigrok_Q4]] by [[User:BryanDavis|BryanDavis]]
* 01:32 [[etherpad:p/Wikidata_technical_needs]] was archived as [[etherpadbackup:Wikidata_technical_needs]] by [[User:BryanDavis|BryanDavis]]
* 01:31 [[etherpad:p/WikiGrokTest1]] was archived as [[etherpadbackup:WikiGrokTest1]] by [[User:BryanDavis|BryanDavis]]
* 01:31 [[etherpad:p/WikiDev16-T114045]] was archived as [[etherpadbackup:WikiDev16-T114045]] by [[User:BryanDavis|BryanDavis]]
* 01:30 [[etherpad:p/Watchlist-20beta-20status]] was archived as [[etherpadbackup:Watchlist-20beta-20status]] by [[User:BryanDavis|BryanDavis]]
* 01:30 [[etherpad:p/WRN202003]] was archived as [[etherpadbackup:WRN202003]] by [[User:BryanDavis|BryanDavis]]
* 01:29 [[etherpad:p/WRN201409]] was archived as [[etherpadbackup:WRN201409]] by [[User:BryanDavis|BryanDavis]]
* 01:29 [[etherpad:p/WRN201408]] was archived as [[etherpadbackup:WRN201408]] by [[User:BryanDavis|BryanDavis]]
* 01:28 [[etherpad:p/WP_BRD_generally]] was archived as [[etherpadbackup:WP_BRD_generally]] by [[User:BryanDavis|BryanDavis]]
* 01:28 [[etherpad:p/WMTC19-T238265]] was archived as [[etherpadbackup:WMTC19-T238265]] by [[User:BryanDavis|BryanDavis]]
* 01:27 [[etherpad:p/WMTC19-T238227]] was archived as [[etherpadbackup:WMTC19-T238227]] by [[User:BryanDavis|BryanDavis]]
* 01:27 [[etherpad:p/WMTC19-T234662]] was archived as [[etherpadbackup:WMTC19-T234662]] by [[User:BryanDavis|BryanDavis]]
* 01:26 [[etherpad:p/WMTC19-T234655]] was archived as [[etherpadbackup:WMTC19-T234655]] by [[User:BryanDavis|BryanDavis]]
* 01:26 [[etherpad:p/WMTC19-T234654]] was archived as [[etherpadbackup:WMTC19-T234654]] by [[User:BryanDavis|BryanDavis]]
* 01:25 [[etherpad:p/WMTC19-T234649]] was archived as [[etherpadbackup:WMTC19-T234649]] by [[User:BryanDavis|BryanDavis]]
* 01:25 [[etherpad:p/WMTC19-T234641]] was archived as [[etherpadbackup:WMTC19-T234641]] by [[User:BryanDavis|BryanDavis]]
* 01:24 [[etherpad:p/WMTC19-T234636]] was archived as [[etherpadbackup:WMTC19-T234636]] by [[User:BryanDavis|BryanDavis]]
* 01:24 [[etherpad:p/WMTC19-T234632]] was archived as [[etherpadbackup:WMTC19-T234632]] by [[User:BryanDavis|BryanDavis]]
* 01:23 [[etherpad:p/WMHack_Feature_Ideas_for_WorkAdventure]] was archived as [[etherpadbackup:WMHack_Feature_Ideas_for_WorkAdventure]] by [[User:BryanDavis|BryanDavis]]
* 01:23 [[etherpad:p/WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] was archived as [[etherpadbackup:WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] by [[User:BryanDavis|BryanDavis]]
* 01:22 [[etherpad:p/WMF_Research_Office_Hours|Notes]] was archived as [[etherpadbackup:WMF_Research_Office_Hours|Notes]] by [[User:BryanDavis|BryanDavis]]
* 01:21 [[etherpad:p/WMCS-techsupport-task]] was archived as [[etherpadbackup:WMCS-techsupport-task]] by [[User:BryanDavis|BryanDavis]]
* 01:21 [[etherpad:p/WMCS-infrafoundations-2021-11-24]] was archived as [[etherpadbackup:WMCS-infrafoundations-2021-11-24]] by [[User:BryanDavis|BryanDavis]]
* 01:20 [[etherpad:p/WMCS-Renaming-Announce]] was archived as [[etherpadbackup:WMCS-Renaming-Announce]] by [[User:BryanDavis|BryanDavis]]
* 01:20 [[etherpad:p/WMCS-PDU-ops-2019-10-24]] was archived as [[etherpadbackup:WMCS-PDU-ops-2019-10-24]] by [[User:BryanDavis|BryanDavis]]
* 01:19 [[etherpad:p/WMCS-2026-02-12]] was archived as [[etherpadbackup:WMCS-2026-02-12]] by [[User:BryanDavis|BryanDavis]]
* 01:19 [[etherpad:p/WMCS-2025-12-18]] was archived as [[etherpadbackup:WMCS-2025-12-18]] by [[User:BryanDavis|BryanDavis]]
* 01:18 [[etherpad:p/WMCS-2025-09-04]] was archived as [[etherpadbackup:WMCS-2025-09-04]] by [[User:BryanDavis|BryanDavis]]
* 01:18 [[etherpad:p/WMCS-2025-08-14]] was archived as [[etherpadbackup:WMCS-2025-08-14]] by [[User:BryanDavis|BryanDavis]]
* 01:17 [[etherpad:p/WMCS-2024-02-07]] was archived as [[etherpadbackup:WMCS-2024-02-07]] by [[User:BryanDavis|BryanDavis]]
* 01:17 [[etherpad:p/WMCS-2023-10-04]] was archived as [[etherpadbackup:WMCS-2023-10-04]] by [[User:BryanDavis|BryanDavis]]
* 01:16 [[etherpad:p/WMCS-2023-09-20]] was archived as [[etherpadbackup:WMCS-2023-09-20]] by [[User:BryanDavis|BryanDavis]]
* 01:16 [[etherpad:p/WMCS-2023-08-10]] was archived as [[etherpadbackup:WMCS-2023-08-10]] by [[User:BryanDavis|BryanDavis]]
* 01:15 [[etherpad:p/WMCS-2023-06-28]] was archived as [[etherpadbackup:WMCS-2023-06-28]] by [[User:BryanDavis|BryanDavis]]
* 01:15 [[etherpad:p/WMCS-2023-04-09-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2023-04-09-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:14 [[etherpad:p/WMCS-2022-11-09]] was archived as [[etherpadbackup:WMCS-2022-11-09]] by [[User:BryanDavis|BryanDavis]]
* 01:14 [[etherpad:p/WMCS-2022-09-07]] was archived as [[etherpadbackup:WMCS-2022-09-07]] by [[User:BryanDavis|BryanDavis]]
* 01:13 [[etherpad:p/WMCS-2022-07-20]] was archived as [[etherpadbackup:WMCS-2022-07-20]] by [[User:BryanDavis|BryanDavis]]
* 01:13 [[etherpad:p/WMCS-2022-07-13]] was archived as [[etherpadbackup:WMCS-2022-07-13]] by [[User:BryanDavis|BryanDavis]]
* 01:12 [[etherpad:p/WMCS-2022-05-11-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2022-05-11-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]]
* 01:12 [[etherpad:p/WMCS-2021-12-15]] was archived as [[etherpadbackup:WMCS-2021-12-15]] by [[User:BryanDavis|BryanDavis]]
* 01:11 [[etherpad:p/WMCS-2021-11-03]] was archived as [[etherpadbackup:WMCS-2021-11-03]] by [[User:BryanDavis|BryanDavis]]
* 01:10 [[etherpad:p/WMCS-2020-12-02]] was archived as [[etherpadbackup:WMCS-2020-12-02]] by [[User:BryanDavis|BryanDavis]]
* 01:10 [[etherpad:p/WMCS-2020-11-25]] was archived as [[etherpadbackup:WMCS-2020-11-25]] by [[User:BryanDavis|BryanDavis]]
* 01:09 [[etherpad:p/WMCS-2020-11-04]] was archived as [[etherpadbackup:WMCS-2020-11-04]] by [[User:BryanDavis|BryanDavis]]
* 01:09 [[etherpad:p/WMCS-2020-04-22]] was archived as [[etherpadbackup:WMCS-2020-04-22]] by [[User:BryanDavis|BryanDavis]]
* 01:08 [[etherpad:p/WMCS-2020-01-07]] was archived as [[etherpadbackup:WMCS-2020-01-07]] by [[User:BryanDavis|BryanDavis]]
* 01:08 [[etherpad:p/WMCS-2019-10-24]] was archived as [[etherpadbackup:WMCS-2019-10-24]] by [[User:BryanDavis|BryanDavis]]
* 01:07 [[etherpad:p/WMCS-2019-09-03]] was archived as [[etherpadbackup:WMCS-2019-09-03]] by [[User:BryanDavis|BryanDavis]]
* 01:07 [[etherpad:p/WMCS-2019-07-30]] was archived as [[etherpadbackup:WMCS-2019-07-30]] by [[User:BryanDavis|BryanDavis]]
* 01:06 [[etherpad:p/WMCS-2019-01-29]] was archived as [[etherpadbackup:WMCS-2019-01-29]] by [[User:BryanDavis|BryanDavis]]
* 01:06 [[etherpad:p/WMCS-2018-10-02]] was archived as [[etherpadbackup:WMCS-2018-10-02]] by [[User:BryanDavis|BryanDavis]]
* 01:05 [[etherpad:p/WMCS-2018-08-14]] was archived as [[etherpadbackup:WMCS-2018-08-14]] by [[User:BryanDavis|BryanDavis]]
* 01:05 [[etherpad:p/WMCS-2018-07-31]] was archived as [[etherpadbackup:WMCS-2018-07-31]] by [[User:BryanDavis|BryanDavis]]
* 01:04 [[etherpad:p/WMCS-2018-07-10]] was archived as [[etherpadbackup:WMCS-2018-07-10]] by [[User:BryanDavis|BryanDavis]]
* 01:04 [[etherpad:p/WMCS-2018-06-19]] was archived as [[etherpadbackup:WMCS-2018-06-19]] by [[User:BryanDavis|BryanDavis]]
* 01:03 [[etherpad:p/WMCS-2018-06-05]] was archived as [[etherpadbackup:WMCS-2018-06-05]] by [[User:BryanDavis|BryanDavis]]
* 01:03 [[etherpad:p/WMCS-2018-05-29]] was archived as [[etherpadbackup:WMCS-2018-05-29]] by [[User:BryanDavis|BryanDavis]]
* 01:02 [[etherpad:p/WMCS-2018-05-01]] was archived as [[etherpadbackup:WMCS-2018-05-01]] by [[User:BryanDavis|BryanDavis]]
* 01:02 [[etherpad:p/WMCS-2018-04-10]] was archived as [[etherpadbackup:WMCS-2018-04-10]] by [[User:BryanDavis|BryanDavis]]
* 01:01 [[etherpad:p/WMCS-2018-04-03]] was archived as [[etherpadbackup:WMCS-2018-04-03]] by [[User:BryanDavis|BryanDavis]]
* 01:01 [[etherpad:p/WMCS-2018-02-13]] was archived as [[etherpadbackup:WMCS-2018-02-13]] by [[User:BryanDavis|BryanDavis]]
* 01:00 [[etherpad:p/WMCS-2018-02-05]] was archived as [[etherpadbackup:WMCS-2018-02-05]] by [[User:BryanDavis|BryanDavis]]
* 01:00 [[etherpad:p/WMCS-2018-01-23]] was archived as [[etherpadbackup:WMCS-2018-01-23]] by [[User:BryanDavis|BryanDavis]]
* 00:59 [[etherpad:p/WMCS-2018-01-09]] was archived as [[etherpadbackup:WMCS-2018-01-09]] by [[User:BryanDavis|BryanDavis]]
* 00:59 [[etherpad:p/WMCS-2017-12-19]] was archived as [[etherpadbackup:WMCS-2017-12-19]] by [[User:BryanDavis|BryanDavis]]
* 00:58 [[etherpad:p/WMCS-2017-11-28]] was archived as [[etherpadbackup:WMCS-2017-11-28]] by [[User:BryanDavis|BryanDavis]]
* 00:58 [[etherpad:p/WMCS-2017-11-21]] was archived as [[etherpadbackup:WMCS-2017-11-21]] by [[User:BryanDavis|BryanDavis]]
* 00:57 [[etherpad:p/WMCS-2017-11-14]] was archived as [[etherpadbackup:WMCS-2017-11-14]] by [[User:BryanDavis|BryanDavis]]
* 00:57 [[etherpad:p/WMCS-2017-11-07]] was archived as [[etherpadbackup:WMCS-2017-11-07]] by [[User:BryanDavis|BryanDavis]]
* 00:56 [[etherpad:p/WMCS-2017-10-30]] was archived as [[etherpadbackup:WMCS-2017-10-30]] by [[User:BryanDavis|BryanDavis]]
* 00:56 [[etherpad:p/WMCS-2017-10-24]] was archived as [[etherpadbackup:WMCS-2017-10-24]] by [[User:BryanDavis|BryanDavis]]
* 00:55 [[etherpad:p/WMCS-2017-10-03]] was archived as [[etherpadbackup:WMCS-2017-10-03]] by [[User:BryanDavis|BryanDavis]]
* 00:54 [[etherpad:p/WMCS-2017-09-19]] was archived as [[etherpadbackup:WMCS-2017-09-19]] by [[User:BryanDavis|BryanDavis]]
* 00:54 [[etherpad:p/WMCS-2017-05-15]] was archived as [[etherpadbackup:WMCS-2017-05-15]] by [[User:BryanDavis|BryanDavis]]
* 00:53 [[etherpad:p/WMCS-2017-05-08]] was archived as [[etherpadbackup:WMCS-2017-05-08]] by [[User:BryanDavis|BryanDavis]]
* 00:53 [[etherpad:p/WMCS-2017-04-17]] was archived as [[etherpadbackup:WMCS-2017-04-17]] by [[User:BryanDavis|BryanDavis]]
* 00:52 [[etherpad:p/WMCS-2017-04-10]] was archived as [[etherpadbackup:WMCS-2017-04-10]] by [[User:BryanDavis|BryanDavis]]
* 00:52 [[etherpad:p/WM2024_Day4_Lviv-_Rooms_21+22+23]] was archived as [[etherpadbackup:WM2024_Day4_Lviv-_Rooms_21+22+23]] by [[User:BryanDavis|BryanDavis]]
* 00:51 [[etherpad:p/WM2024_Day1_Warsaw-_Rooms_20+24]] was archived as [[etherpadbackup:WM2024_Day1_Warsaw-_Rooms_20+24]] by [[User:BryanDavis|BryanDavis]]
* 00:51 [[etherpad:p/WLMshowcase]] was archived as [[etherpadbackup:WLMshowcase]] by [[User:BryanDavis|BryanDavis]]
* 00:50 [[etherpad:p/WLMShowcase2]] was archived as [[etherpadbackup:WLMShowcase2]] by [[User:BryanDavis|BryanDavis]]
* 00:50 [[etherpad:p/WLMMobile]] was archived as [[etherpadbackup:WLMMobile]] by [[User:BryanDavis|BryanDavis]]
* 00:49 [[etherpad:p/WLMMarketToDo]] was archived as [[etherpadbackup:WLMMarketToDo]] by [[User:BryanDavis|BryanDavis]]
* 00:49 [[etherpad:p/WLMAppEmail]] was archived as [[etherpadbackup:WLMAppEmail]] by [[User:BryanDavis|BryanDavis]]
* 00:48 [[etherpad:p/WLM-on-cluster]] was archived as [[etherpadbackup:WLM-on-cluster]] by [[User:BryanDavis|BryanDavis]]
* 00:48 [[etherpad:p/WLM-mobile]] was archived as [[etherpadbackup:WLM-mobile]] by [[User:BryanDavis|BryanDavis]]
* 00:47 [[etherpad:p/WLM-API-to-WMF]] was archived as [[etherpadbackup:WLM-API-to-WMF]] by [[User:BryanDavis|BryanDavis]]
* 00:47 [[etherpad:p/WLM-2013-07-05]] was archived as [[etherpadbackup:WLM-2013-07-05]] by [[User:BryanDavis|BryanDavis]]
* 00:46 [[etherpad:p/VzkZSy0PKK]] was archived as [[etherpadbackup:VzkZSy0PKK]] by [[User:BryanDavis|BryanDavis]]
* 00:46 [[etherpad:p/VisualEditorTeamLunchOrder20140327]] was archived as [[etherpadbackup:VisualEditorTeamLunchOrder20140327]] by [[User:BryanDavis|BryanDavis]]
* 00:45 [[etherpad:p/VelocityNYC2015-Day1]] was archived as [[etherpadbackup:VelocityNYC2015-Day1]] by [[User:BryanDavis|BryanDavis]]
* 00:45 [[etherpad:p/Vagrant-wannabees]] was archived as [[etherpadbackup:Vagrant-wannabees]] by [[User:BryanDavis|BryanDavis]]
* 00:44 [[etherpad:p/VPT]] was archived as [[etherpadbackup:VPT]] by [[User:BryanDavis|BryanDavis]]
* 00:44 [[etherpad:p/VM_puppet_failures]] was archived as [[etherpadbackup:VM_puppet_failures]] by [[User:BryanDavis|BryanDavis]]
* 00:43 [[etherpad:p/VM_puppet_disabled]] was archived as [[etherpadbackup:VM_puppet_disabled]] by [[User:BryanDavis|BryanDavis]]
* 00:43 [[etherpad:p/VE_bugs]] was archived as [[etherpadbackup:VE_bugs]] by [[User:BryanDavis|BryanDavis]]
* 00:42 [[etherpad:p/VETOC]] was archived as [[etherpadbackup:VETOC]] by [[User:BryanDavis|BryanDavis]]
* 00:42 [[etherpad:p/VEQ3Perf]] was archived as [[etherpadbackup:VEQ3Perf]] by [[User:BryanDavis|BryanDavis]]
* 00:41 [[etherpad:p/VE-Chromium-Upstream]] was archived as [[etherpadbackup:VE-Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]]
* 00:41 [[etherpad:p/UwSprintAugust2012]] was archived as [[etherpadbackup:UwSprintAugust2012]] by [[User:BryanDavis|BryanDavis]]
* 00:40 [[etherpad:p/Upload_debugging]] was archived as [[etherpadbackup:Upload_debugging]] by [[User:BryanDavis|BryanDavis]]
* 00:39 [[etherpad:p/UploadWizard-and-EventLogging]] was archived as [[etherpadbackup:UploadWizard-and-EventLogging]] by [[User:BryanDavis|BryanDavis]]
* 00:39 [[etherpad:p/Upgrade-now-bitches]] was archived as [[etherpadbackup:Upgrade-now-bitches]] by [[User:BryanDavis|BryanDavis]]
* 00:38 [[etherpad:p/UniqueServiceGroups]] was archived as [[etherpadbackup:UniqueServiceGroups]] by [[User:BryanDavis|BryanDavis]]
* 00:38 [[etherpad:p/Udeme_Emmanson]] was archived as [[etherpadbackup:Udeme_Emmanson]] by [[User:BryanDavis|BryanDavis]]
* 00:37 [[etherpad:p/UShQLRyUMv]] was archived as [[etherpadbackup:UShQLRyUMv]] by [[User:BryanDavis|BryanDavis]]
* 00:37 [[etherpad:p/UA-SchemaUpdate]] was archived as [[etherpadbackup:UA-SchemaUpdate]] by [[User:BryanDavis|BryanDavis]]
* 00:36 [[etherpad:p/U84-IdW1WfD2vZ4ogSCx]] was archived as [[etherpadbackup:U84-IdW1WfD2vZ4ogSCx]] by [[User:BryanDavis|BryanDavis]]
* 00:36 [[etherpad:p/U53g0zdCtb]] was archived as [[etherpadbackup:U53g0zdCtb]] by [[User:BryanDavis|BryanDavis]]
* 00:35 [[etherpad:p/U]] was archived as [[etherpadbackup:U]] by [[User:BryanDavis|BryanDavis]]
* 00:35 [[etherpad:p/Trusty_grid_shutdown]] was archived as [[etherpadbackup:Trusty_grid_shutdown]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/TransclusionModelIntroduction]] was archived as [[etherpadbackup:TransclusionModelIntroduction]] by [[User:BryanDavis|BryanDavis]]
* 00:34 [[etherpad:p/Traffic-2021-08-24]] was archived as [[etherpadbackup:Traffic-2021-08-24]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/Traffic-2021-08-17]] was archived as [[etherpadbackup:Traffic-2021-08-17]] by [[User:BryanDavis|BryanDavis]]
* 00:33 [[etherpad:p/Traffic-2021-02-17]] was archived as [[etherpadbackup:Traffic-2021-02-17]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/Traffic-2020-12-16]] was archived as [[etherpadbackup:Traffic-2020-12-16]] by [[User:BryanDavis|BryanDavis]]
* 00:32 [[etherpad:p/Traffic-2020-11-02]] was archived as [[etherpadbackup:Traffic-2020-11-02]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/Traffic-2020-10-13]] was archived as [[etherpadbackup:Traffic-2020-10-13]] by [[User:BryanDavis|BryanDavis]]
* 00:31 [[etherpad:p/Traffic-2020-09-28]] was archived as [[etherpadbackup:Traffic-2020-09-28]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/Traffic-2018-06-28]] was archived as [[etherpadbackup:Traffic-2018-06-28]] by [[User:BryanDavis|BryanDavis]]
* 00:30 [[etherpad:p/Traffic-2018-06-14]] was archived as [[etherpadbackup:Traffic-2018-06-14]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/ToolLabs_Survey_feedback]] was archived as [[etherpadbackup:ToolLabs_Survey_feedback]] by [[User:BryanDavis|BryanDavis]]
* 00:29 [[etherpad:p/TextInputWidgetValidation]] was archived as [[etherpadbackup:TextInputWidgetValidation]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/Templates]] was archived as [[etherpadbackup:Templates]] by [[User:BryanDavis|BryanDavis]]
* 00:28 [[etherpad:p/Technical_Engagment_All_Hands_2019doing_and_should]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019doing_and_should]] by [[User:BryanDavis|BryanDavis]]
* 00:27 [[etherpad:p/Technical_Engagment_All_Hands_2019]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019]] by [[User:BryanDavis|BryanDavis]]
* 00:27 [[etherpad:p/TechOps-goals-Q4-FY15-16]] was archived as [[etherpadbackup:TechOps-goals-Q4-FY15-16]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/TechOps-goals-FQ3-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ3-FY1617]] by [[User:BryanDavis|BryanDavis]]
* 00:26 [[etherpad:p/TechOps-goals-FQ2-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ2-FY1617]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/TechOps-goals-FQ1-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ1-FY1617]] by [[User:BryanDavis|BryanDavis]]
* 00:25 [[etherpad:p/TechOps-2017-09-18]] was archived as [[etherpadbackup:TechOps-2017-09-18]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/TechOps-2015-10-14]] was archived as [[etherpadbackup:TechOps-2015-10-14]] by [[User:BryanDavis|BryanDavis]]
* 00:24 [[etherpad:p/TechOps-2015-07-27]] was archived as [[etherpadbackup:TechOps-2015-07-27]] by [[User:BryanDavis|BryanDavis]]
* 00:23 [[etherpad:p/TechDaysCareers]] was archived as [[etherpadbackup:TechDaysCareers]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/TechDays-Mobile]] was archived as [[etherpadbackup:TechDays-Mobile]] by [[User:BryanDavis|BryanDavis]]
* 00:22 [[etherpad:p/TargetLoader]] was archived as [[etherpadbackup:TargetLoader]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/T_movie]] was archived as [[etherpadbackup:T_movie]] by [[User:BryanDavis|BryanDavis]]
* 00:21 [[etherpad:p/TFMQF]] was archived as [[etherpadbackup:TFMQF]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/T404584-announcement]] was archived as [[etherpadbackup:T404584-announcement]] by [[User:BryanDavis|BryanDavis]]
* 00:20 [[etherpad:p/T395855]] was archived as [[etherpadbackup:T395855]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/T372912]] was archived as [[etherpadbackup:T372912]] by [[User:BryanDavis|BryanDavis]]
* 00:19 [[etherpad:p/T364459]] was archived as [[etherpadbackup:T364459]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/T340241-new-fingerprints]] was archived as [[etherpadbackup:T340241-new-fingerprints]] by [[User:BryanDavis|BryanDavis]]
* 00:18 [[etherpad:p/T340241-new-bastion]] was archived as [[etherpadbackup:T340241-new-bastion]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/T221119_-_log]] was archived as [[etherpadbackup:T221119_-_log]] by [[User:BryanDavis|BryanDavis]]
* 00:17 [[etherpad:p/T191532]] was archived as [[etherpadbackup:T191532]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/T182722-outage]] was archived as [[etherpadbackup:T182722-outage]] by [[User:BryanDavis|BryanDavis]]
* 00:16 [[etherpad:p/T152122_notes]] was archived as [[etherpadbackup:T152122_notes]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/T128190]] was archived as [[etherpadbackup:T128190]] by [[User:BryanDavis|BryanDavis]]
* 00:15 [[etherpad:p/Swift-Switch-Originals]] was archived as [[etherpadbackup:Swift-Switch-Originals]] by [[User:BryanDavis|BryanDavis]]
* 00:14 [[etherpad:p/Summary/_Hypothesis]] was archived as [[etherpadbackup:Summary/_Hypothesis]] by [[User:BryanDavis|BryanDavis]]
* 00:14 [[etherpad:p/SquidToVarnish]] was archived as [[etherpadbackup:SquidToVarnish]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/Sqoop]] was archived as [[etherpadbackup:Sqoop]] by [[User:BryanDavis|BryanDavis]]
* 00:13 [[etherpad:p/Sprint_D]] was archived as [[etherpadbackup:Sprint_D]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/ShutdownPreciseGrid]] was archived as [[etherpadbackup:ShutdownPreciseGrid]] by [[User:BryanDavis|BryanDavis]]
* 00:12 [[etherpad:p/ServiceOps-AugustReboots]] was archived as [[etherpadbackup:ServiceOps-AugustReboots]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/ServiceOps]] was archived as [[etherpadbackup:ServiceOps]] by [[User:BryanDavis|BryanDavis]]
* 00:11 [[etherpad:p/Searchfdbksurvey]] was archived as [[etherpadbackup:Searchfdbksurvey]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/Search_UserStories]] was archived as [[etherpadbackup:Search_UserStories]] by [[User:BryanDavis|BryanDavis]]
* 00:10 [[etherpad:p/ScrummasterMeetup-2013-10-10]] was archived as [[etherpadbackup:ScrummasterMeetup-2013-10-10]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/Scrum-of-Scrums-copypaste]] was archived as [[etherpadbackup:Scrum-of-Scrums-copypaste]] by [[User:BryanDavis|BryanDavis]]
* 00:09 [[etherpad:p/SandboxHatesMe]] was archived as [[etherpadbackup:SandboxHatesMe]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/SWAT_2016-04-14]] was archived as [[etherpadbackup:SWAT_2016-04-14]] by [[User:BryanDavis|BryanDavis]]
* 00:08 [[etherpad:p/SWAT]] was archived as [[etherpadbackup:SWAT]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/SVImxzLIyw]] was archived as [[etherpadbackup:SVImxzLIyw]] by [[User:BryanDavis|BryanDavis]]
* 00:07 [[etherpad:p/SRE-goals-FQ4-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ4-FY1819]] by [[User:BryanDavis|BryanDavis]]
* 00:06 [[etherpad:p/SRE-goals-FQ3-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ3-FY1819]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/SRE-goals-FQ1-FY1920]] was archived as [[etherpadbackup:SRE-goals-FQ1-FY1920]] by [[User:BryanDavis|BryanDavis]]
* 00:05 [[etherpad:p/SRE-ServiceOps-StatusMeeting]] was archived as [[etherpadbackup:SRE-ServiceOps-StatusMeeting]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/SRE-ServiceOps-2019-01-03]] was archived as [[etherpadbackup:SRE-ServiceOps-2019-01-03]] by [[User:BryanDavis|BryanDavis]]
* 00:04 [[etherpad:p/SRE-Foundations-2021-10-25]] was archived as [[etherpadbackup:SRE-Foundations-2021-10-25]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/SRE-Foundations-2020-07-01]] was archived as [[etherpadbackup:SRE-Foundations-2020-07-01]] by [[User:BryanDavis|BryanDavis]]
* 00:03 [[etherpad:p/SRE-Foundations-2020-03-11]] was archived as [[etherpadbackup:SRE-Foundations-2020-03-11]] by [[User:BryanDavis|BryanDavis]]
* 00:02 [[etherpad:p/SRE-Foundations-2020-02-12]] was archived as [[etherpadbackup:SRE-Foundations-2020-02-12]] by [[User:BryanDavis|BryanDavis]]
* 00:02 [[etherpad:p/SRE-Foundations-2019-11-13]] was archived as [[etherpadbackup:SRE-Foundations-2019-11-13]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/SRE-Foundations-2019-08-07]] was archived as [[etherpadbackup:SRE-Foundations-2019-08-07]] by [[User:BryanDavis|BryanDavis]]
* 00:01 [[etherpad:p/SRE-Foundations-2019-07-31]] was archived as [[etherpadbackup:SRE-Foundations-2019-07-31]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/SOAAuth]] was archived as [[etherpadbackup:SOAAuth]] by [[User:BryanDavis|BryanDavis]]
* 00:00 [[etherpad:p/SGE_rebuild]] was archived as [[etherpadbackup:SGE_rebuild]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-05 ===
* 23:59 [[etherpad:p/S]] was archived as [[etherpadbackup:S]] by [[User:BryanDavis|BryanDavis]]
* 23:59 [[etherpad:p/RobotsMetaTag]] was archived as [[etherpadbackup:RobotsMetaTag]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/ResourceLoader-for-mobile]] was archived as [[etherpadbackup:ResourceLoader-for-mobile]] by [[User:BryanDavis|BryanDavis]]
* 23:58 [[etherpad:p/ResourceLoader-Mobile]] was archived as [[etherpadbackup:ResourceLoader-Mobile]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/ResourceLoader]] was archived as [[etherpadbackup:ResourceLoader]] by [[User:BryanDavis|BryanDavis]]
* 23:57 [[etherpad:p/ReleaseNotes]] was archived as [[etherpadbackup:ReleaseNotes]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/ReferenceInspector]] was archived as [[etherpadbackup:ReferenceInspector]] by [[User:BryanDavis|BryanDavis]]
* 23:56 [[etherpad:p/Reading_web_sprint_2_names]] was archived as [[etherpadbackup:Reading_web_sprint_2_names]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/ReadingWebQ3Planning]] was archived as [[etherpadbackup:ReadingWebQ3Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:55 [[etherpad:p/ReadingWebCodeReviewSessions]] was archived as [[etherpadbackup:ReadingWebCodeReviewSessions]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/ReadingShowcase]] was archived as [[etherpadbackup:ReadingShowcase]] by [[User:BryanDavis|BryanDavis]]
* 23:54 [[etherpad:p/Reading]] was archived as [[etherpadbackup:Reading]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/RdvI3eLIj5AMM4PN_hHz]] was archived as [[etherpadbackup:RdvI3eLIj5AMM4PN_hHz]] by [[User:BryanDavis|BryanDavis]]
* 23:53 [[etherpad:p/RWW-WikipediaZero]] was archived as [[etherpadbackup:RWW-WikipediaZero]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/RLRLRL]] was archived as [[etherpadbackup:RLRLRL]] by [[User:BryanDavis|BryanDavis]]
* 23:52 [[etherpad:p/RLFuncDeps]] was archived as [[etherpadbackup:RLFuncDeps]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/RLExceptions]] was archived as [[etherpadbackup:RLExceptions]] by [[User:BryanDavis|BryanDavis]]
* 23:51 [[etherpad:p/RL-mwloader-testcases-tmp]] was archived as [[etherpadbackup:RL-mwloader-testcases-tmp]] by [[User:BryanDavis|BryanDavis]]
* 23:50 [[etherpad:p/RL-mwloader-testcases]] was archived as [[etherpadbackup:RL-mwloader-testcases]] by [[User:BryanDavis|BryanDavis]]
* 23:50 [[etherpad:p/RL-async-head]] was archived as [[etherpadbackup:RL-async-head]] by [[User:BryanDavis|BryanDavis]]
* 23:49 [[etherpad:p/RD201412]] was archived as [[etherpadbackup:RD201412]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/RD201411]] was archived as [[etherpadbackup:RD201411]] by [[User:BryanDavis|BryanDavis]]
* 23:48 [[etherpad:p/RD201409]] was archived as [[etherpadbackup:RD201409]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/RD2014]] was archived as [[etherpadbackup:RD2014]] by [[User:BryanDavis|BryanDavis]]
* 23:47 [[etherpad:p/Quarterly_metrics_scorecard]] was archived as [[etherpadbackup:Quarterly_metrics_scorecard]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/QgiZvx5mWMQzlM03zCC1]] was archived as [[etherpadbackup:QgiZvx5mWMQzlM03zCC1]] by [[User:BryanDavis|BryanDavis]]
* 23:46 [[etherpad:p/QR1]] was archived as [[etherpadbackup:QR1]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/QPP]] was archived as [[etherpadbackup:QPP]] by [[User:BryanDavis|BryanDavis]]
* 23:45 [[etherpad:p/QA_Automation_sync_up]] was archived as [[etherpadbackup:QA_Automation_sync_up]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/Q]] was archived as [[etherpadbackup:Q]] by [[User:BryanDavis|BryanDavis]]
* 23:44 [[etherpad:p/Puppet3]] was archived as [[etherpadbackup:Puppet3]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/ProofreadPageLua]] was archived as [[etherpadbackup:ProofreadPageLua]] by [[User:BryanDavis|BryanDavis]]
* 23:43 [[etherpad:p/Phragile-requirements-2015-05]] was archived as [[etherpadbackup:Phragile-requirements-2015-05]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/Phabricator_Upgrade_Planning_20190123]] was archived as [[etherpadbackup:Phabricator_Upgrade_Planning_20190123]] by [[User:BryanDavis|BryanDavis]]
* 23:42 [[etherpad:p/PhabricatorInstall]] was archived as [[etherpadbackup:PhabricatorInstall]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/PerformanceUpdate201601]] was archived as [[etherpadbackup:PerformanceUpdate201601]] by [[User:BryanDavis|BryanDavis]]
* 23:41 [[etherpad:p/PerformanceDecemberUpdate]] was archived as [[etherpadbackup:PerformanceDecemberUpdate]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/Performance-2015-12-14]] was archived as [[etherpadbackup:Performance-2015-12-14]] by [[User:BryanDavis|BryanDavis]]
* 23:40 [[etherpad:p/Performance]] was archived as [[etherpadbackup:Performance]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/PartnerEngineerTraining]] was archived as [[etherpadbackup:PartnerEngineerTraining]] by [[User:BryanDavis|BryanDavis]]
* 23:39 [[etherpad:p/PV_Analytics_Infra]] was archived as [[etherpadbackup:PV_Analytics_Infra]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/PVTransition]] was archived as [[etherpadbackup:PVTransition]] by [[User:BryanDavis|BryanDavis]]
* 23:38 [[etherpad:p/PAWS-2020-04-28]] was archived as [[etherpadbackup:PAWS-2020-04-28]] by [[User:BryanDavis|BryanDavis]]
* 23:37 [[etherpad:p/P&amp;EDashboard20160112]] was archived as [[etherpadbackup:P&amp;EDashboard20160112]] by [[User:BryanDavis|BryanDavis]]
* 23:37 [[etherpad:p/P]] was archived as [[etherpadbackup:P]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/OsJfzQH80T]] was archived as [[etherpadbackup:OsJfzQH80T]] by [[User:BryanDavis|BryanDavis]]
* 23:36 [[etherpad:p/OpsSummit2013Day2]] was archived as [[etherpadbackup:OpsSummit2013Day2]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/OldWmfBranchCleanup]] was archived as [[etherpadbackup:OldWmfBranchCleanup]] by [[User:BryanDavis|BryanDavis]]
* 23:35 [[etherpad:p/OctoberMetricsAgenda]] was archived as [[etherpadbackup:OctoberMetricsAgenda]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/OSM-Wikimedia]] was archived as [[etherpadbackup:OSM-Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 23:34 [[etherpad:p/OOjsUI-TODO]] was archived as [[etherpadbackup:OOjsUI-TODO]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/NewMobileTech]] was archived as [[etherpadbackup:NewMobileTech]] by [[User:BryanDavis|BryanDavis]]
* 23:33 [[etherpad:p/NewDeploysToday]] was archived as [[etherpadbackup:NewDeploysToday]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/N_sprint]] was archived as [[etherpadbackup:N_sprint]] by [[User:BryanDavis|BryanDavis]]
* 23:32 [[etherpad:p/NQpvqJeiKg]] was archived as [[etherpadbackup:NQpvqJeiKg]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/N5q9EmRswClmLQyBOezv]] was archived as [[etherpadbackup:N5q9EmRswClmLQyBOezv]] by [[User:BryanDavis|BryanDavis]]
* 23:31 [[etherpad:p/Mobile_at_Wikimania_2015]] was archived as [[etherpadbackup:Mobile_at_Wikimania_2015]] by [[User:BryanDavis|BryanDavis]]
* 23:30 [[etherpad:p/Mobile_Parsoid]] was archived as [[etherpadbackup:Mobile_Parsoid]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/MobileWeb_Iteration_name]] was archived as [[etherpadbackup:MobileWeb_Iteration_name]] by [[User:BryanDavis|BryanDavis]]
* 23:29 [[etherpad:p/MobileWebOnboarding]] was archived as [[etherpadbackup:MobileWebOnboarding]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/MobileWebDevelopment]] was archived as [[etherpadbackup:MobileWebDevelopment]] by [[User:BryanDavis|BryanDavis]]
* 23:28 [[etherpad:p/MobileWeb-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Retrospective]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/MobileWeb-Q4-Planning]] was archived as [[etherpadbackup:MobileWeb-Q4-Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:27 [[etherpad:p/MobileWeb-Q3-Planning]] was archived as [[etherpadbackup:MobileWeb-Q3-Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:26 [[etherpad:p/MobileWeb-Q2-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Q2-Retrospective]] by [[User:BryanDavis|BryanDavis]]
* 23:26 [[etherpad:p/MobileWeb-IterationName]] was archived as [[etherpadbackup:MobileWeb-IterationName]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/MobileWeb-CodeReview-Discussion]] was archived as [[etherpadbackup:MobileWeb-CodeReview-Discussion]] by [[User:BryanDavis|BryanDavis]]
* 23:25 [[etherpad:p/MobileWatchlist]] was archived as [[etherpadbackup:MobileWatchlist]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/MobileUX]] was archived as [[etherpadbackup:MobileUX]] by [[User:BryanDavis|BryanDavis]]
* 23:24 [[etherpad:p/MobileTutorial]] was archived as [[etherpadbackup:MobileTutorial]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/MobileToCore]] was archived as [[etherpadbackup:MobileToCore]] by [[User:BryanDavis|BryanDavis]]
* 23:23 [[etherpad:p/MobileTechDays]] was archived as [[etherpadbackup:MobileTechDays]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/MobileTeamLunchHangout]] was archived as [[etherpadbackup:MobileTeamLunchHangout]] by [[User:BryanDavis|BryanDavis]]
* 23:22 [[etherpad:p/MobileTeamHangout]] was archived as [[etherpadbackup:MobileTeamHangout]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/MobileShowcase]] was archived as [[etherpadbackup:MobileShowcase]] by [[User:BryanDavis|BryanDavis]]
* 23:21 [[etherpad:p/MobileResourcePlanning]] was archived as [[etherpadbackup:MobileResourcePlanning]] by [[User:BryanDavis|BryanDavis]]
* 23:20 [[etherpad:p/MobileQ1_2014_2015]] was archived as [[etherpadbackup:MobileQ1_2014_2015]] by [[User:BryanDavis|BryanDavis]]
* 23:20 [[etherpad:p/MobileProjectPlanning-Sept2012]] was archived as [[etherpadbackup:MobileProjectPlanning-Sept2012]] by [[User:BryanDavis|BryanDavis]]
* 23:19 [[etherpad:p/MobilePageLoadStats]] was archived as [[etherpadbackup:MobilePageLoadStats]] by [[User:BryanDavis|BryanDavis]]
* 23:19 [[etherpad:p/MobileOps]] was archived as [[etherpadbackup:MobileOps]] by [[User:BryanDavis|BryanDavis]]
* 23:18 [[etherpad:p/MobileOperatingSystems]] was archived as [[etherpadbackup:MobileOperatingSystems]] by [[User:BryanDavis|BryanDavis]]
* 23:18 [[etherpad:p/MobileFrontendCore]] was archived as [[etherpadbackup:MobileFrontendCore]] by [[User:BryanDavis|BryanDavis]]
* 23:17 [[etherpad:p/MobileFrontendChanges]] was archived as [[etherpadbackup:MobileFrontendChanges]] by [[User:BryanDavis|BryanDavis]]
* 23:17 [[etherpad:p/MobileFrontend-core]] was archived as [[etherpadbackup:MobileFrontend-core]] by [[User:BryanDavis|BryanDavis]]
* 23:16 [[etherpad:p/MobileFrontend-Deployment-Issues-20120423]] was archived as [[etherpadbackup:MobileFrontend-Deployment-Issues-20120423]] by [[User:BryanDavis|BryanDavis]]
* 23:16 [[etherpad:p/MobileFrontend-20120507-deployment-issues]] was archived as [[etherpadbackup:MobileFrontend-20120507-deployment-issues]] by [[User:BryanDavis|BryanDavis]]
* 23:15 [[etherpad:p/MobileDeploymentPainPoints]] was archived as [[etherpadbackup:MobileDeploymentPainPoints]] by [[User:BryanDavis|BryanDavis]]
* 23:15 [[etherpad:p/MobileChangeLog]] was archived as [[etherpadbackup:MobileChangeLog]] by [[User:BryanDavis|BryanDavis]]
* 23:14 [[etherpad:p/MobileAsks]] was archived as [[etherpadbackup:MobileAsks]] by [[User:BryanDavis|BryanDavis]]
* 23:13 [[etherpad:p/MobileArchitectureReview]] was archived as [[etherpadbackup:MobileArchitectureReview]] by [[User:BryanDavis|BryanDavis]]
* 23:13 [[etherpad:p/MobileAppTeam]] was archived as [[etherpadbackup:MobileAppTeam]] by [[User:BryanDavis|BryanDavis]]
* 23:12 [[etherpad:p/MobileAppRetrospective]] was archived as [[etherpadbackup:MobileAppRetrospective]] by [[User:BryanDavis|BryanDavis]]
* 23:12 [[etherpad:p/MobileAppReleaseManger]] was archived as [[etherpadbackup:MobileAppReleaseManger]] by [[User:BryanDavis|BryanDavis]]
* 23:11 [[etherpad:p/Mobile23Oct12]] was archived as [[etherpadbackup:Mobile23Oct12]] by [[User:BryanDavis|BryanDavis]]
* 23:11 [[etherpad:p/Mobile-WMDS]] was archived as [[etherpadbackup:Mobile-WMDS]] by [[User:BryanDavis|BryanDavis]]
* 23:10 [[etherpad:p/Mobile-VE-Integration-Planning]] was archived as [[etherpadbackup:Mobile-VE-Integration-Planning]] by [[User:BryanDavis|BryanDavis]]
* 23:10 [[etherpad:p/Mobile-RL]] was archived as [[etherpadbackup:Mobile-RL]] by [[User:BryanDavis|BryanDavis]]
* 23:09 [[etherpad:p/Mobile-20navigation-20planning]] was archived as [[etherpadbackup:Mobile-20navigation-20planning]] by [[User:BryanDavis|BryanDavis]]
* 23:09 [[etherpad:p/Mm3uNugLtk]] was archived as [[etherpadbackup:Mm3uNugLtk]] by [[User:BryanDavis|BryanDavis]]
* 23:08 [[etherpad:p/Metrics_page]] was archived as [[etherpadbackup:Metrics_page]] by [[User:BryanDavis|BryanDavis]]
* 23:08 [[etherpad:p/Meetup]] was archived as [[etherpadbackup:Meetup]] by [[User:BryanDavis|BryanDavis]]
* 23:07 [[etherpad:p/MediaWiki_Workshop_Buea_-_Participants_emails]] was archived as [[etherpadbackup:MediaWiki_Workshop_Buea_-_Participants_emails]] by [[User:BryanDavis|BryanDavis]]
* 23:07 [[etherpad:p/MediaWiki_UI_Standardization_hack4]] was archived as [[etherpadbackup:MediaWiki_UI_Standardization_hack4]] by [[User:BryanDavis|BryanDavis]]
* 23:06 [[etherpad:p/McUX]] was archived as [[etherpadbackup:McUX]] by [[User:BryanDavis|BryanDavis]]
* 23:06 [[etherpad:p/Maps_pad]] was archived as [[etherpadbackup:Maps_pad]] by [[User:BryanDavis|BryanDavis]]
* 23:05 [[etherpad:p/MWDS2015_multi-device_world]] was archived as [[etherpadbackup:MWDS2015_multi-device_world]] by [[User:BryanDavis|BryanDavis]]
* 23:05 [[etherpad:p/MWDS2015_developer_documentation]] was archived as [[etherpadbackup:MWDS2015_developer_documentation]] by [[User:BryanDavis|BryanDavis]]
* 23:04 [[etherpad:p/MWCore2015Jan]] was archived as [[etherpadbackup:MWCore2015Jan]] by [[User:BryanDavis|BryanDavis]]
* 23:04 [[etherpad:p/MFCommitGuidelines]] was archived as [[etherpadbackup:MFCommitGuidelines]] by [[User:BryanDavis|BryanDavis]]
* 23:03 [[etherpad:p/MF-betalabs-planning]] was archived as [[etherpadbackup:MF-betalabs-planning]] by [[User:BryanDavis|BryanDavis]]
* 23:03 [[etherpad:p/MDS_2015_SOA_plenary]] was archived as [[etherpadbackup:MDS_2015_SOA_plenary]] by [[User:BryanDavis|BryanDavis]]
* 23:02 [[etherpad:p/MDS_2015_SOA_and_operations]] was archived as [[etherpadbackup:MDS_2015_SOA_and_operations]] by [[User:BryanDavis|BryanDavis]]
* 23:02 [[etherpad:p/Luke081515]] was archived as [[etherpadbackup:Luke081515]] by [[User:BryanDavis|BryanDavis]]
* 23:01 [[etherpad:p/Logout_Confirmation_DRC]] was archived as [[etherpadbackup:Logout_Confirmation_DRC]] by [[User:BryanDavis|BryanDavis]]
* 23:01 [[etherpad:p/LibrarizationRFC]] was archived as [[etherpadbackup:LibrarizationRFC]] by [[User:BryanDavis|BryanDavis]]
* 23:00 [[etherpad:p/Libera-channel-queue]] was archived as [[etherpadbackup:Libera-channel-queue]] by [[User:BryanDavis|BryanDavis]]
* 23:00 [[etherpad:p/Legends_of_Lavanya]] was archived as [[etherpadbackup:Legends_of_Lavanya]] by [[User:BryanDavis|BryanDavis]]
* 22:59 [[etherpad:p/LabsUpgradePlans]] was archived as [[etherpadbackup:LabsUpgradePlans]] by [[User:BryanDavis|BryanDavis]]
* 22:59 [[etherpad:p/LabsIrcConf]] was archived as [[etherpadbackup:LabsIrcConf]] by [[User:BryanDavis|BryanDavis]]
* 22:58 [[etherpad:p/LabsAPI]] was archived as [[etherpadbackup:LabsAPI]] by [[User:BryanDavis|BryanDavis]]
* 22:58 [[etherpad:p/Labs-Incident-Report-20015-03-31]] was archived as [[etherpadbackup:Labs-Incident-Report-20015-03-31]] by [[User:BryanDavis|BryanDavis]]
* 22:57 [[etherpad:p/Labs-2017-03-14]] was archived as [[etherpadbackup:Labs-2017-03-14]] by [[User:BryanDavis|BryanDavis]]
* 22:57 [[etherpad:p/Labs-2017-03-06]] was archived as [[etherpadbackup:Labs-2017-03-06]] by [[User:BryanDavis|BryanDavis]]
* 22:56 [[etherpad:p/Labs-2017-02-22]] was archived as [[etherpadbackup:Labs-2017-02-22]] by [[User:BryanDavis|BryanDavis]]
* 22:56 [[etherpad:p/Labs-2017-01-23]] was archived as [[etherpadbackup:Labs-2017-01-23]] by [[User:BryanDavis|BryanDavis]]
* 22:55 [[etherpad:p/Labs-2016-12-12]] was archived as [[etherpadbackup:Labs-2016-12-12]] by [[User:BryanDavis|BryanDavis]]
* 22:54 [[etherpad:p/Labs-2016-11-07]] was archived as [[etherpadbackup:Labs-2016-11-07]] by [[User:BryanDavis|BryanDavis]]
* 22:54 [[etherpad:p/Labs-2016-10-31]] was archived as [[etherpadbackup:Labs-2016-10-31]] by [[User:BryanDavis|BryanDavis]]
* 22:53 [[etherpad:p/Labs-2016-09-19]] was archived as [[etherpadbackup:Labs-2016-09-19]] by [[User:BryanDavis|BryanDavis]]
* 22:53 [[etherpad:p/Labs-2016-09-06]] was archived as [[etherpadbackup:Labs-2016-09-06]] by [[User:BryanDavis|BryanDavis]]
* 22:52 [[etherpad:p/Labs-2016-08-29]] was archived as [[etherpadbackup:Labs-2016-08-29]] by [[User:BryanDavis|BryanDavis]]
* 22:52 [[etherpad:p/Labs-2016-08-01]] was archived as [[etherpadbackup:Labs-2016-08-01]] by [[User:BryanDavis|BryanDavis]]
* 22:51 [[etherpad:p/Labs-2016-07-25]] was archived as [[etherpadbackup:Labs-2016-07-25]] by [[User:BryanDavis|BryanDavis]]
* 22:51 [[etherpad:p/Labs-2016-07-11]] was archived as [[etherpadbackup:Labs-2016-07-11]] by [[User:BryanDavis|BryanDavis]]
* 22:50 [[etherpad:p/Labs-2016-06-06]] was archived as [[etherpadbackup:Labs-2016-06-06]] by [[User:BryanDavis|BryanDavis]]
* 22:50 [[etherpad:p/Labs-2016-05-23]] was archived as [[etherpadbackup:Labs-2016-05-23]] by [[User:BryanDavis|BryanDavis]]
* 22:49 [[etherpad:p/Labs-2016-05-16]] was archived as [[etherpadbackup:Labs-2016-05-16]] by [[User:BryanDavis|BryanDavis]]
* 22:49 [[etherpad:p/Labs-2016-05-09]] was archived as [[etherpadbackup:Labs-2016-05-09]] by [[User:BryanDavis|BryanDavis]]
* 22:48 [[etherpad:p/Labs-2016-05-02]] was archived as [[etherpadbackup:Labs-2016-05-02]] by [[User:BryanDavis|BryanDavis]]
* 22:48 [[etherpad:p/Labs-2016-04-25]] was archived as [[etherpadbackup:Labs-2016-04-25]] by [[User:BryanDavis|BryanDavis]]
* 22:47 [[etherpad:p/Labs-20150602]] was archived as [[etherpadbackup:Labs-20150602]] by [[User:BryanDavis|BryanDavis]]
* 22:47 [[etherpad:p/L]] was archived as [[etherpadbackup:L]] by [[User:BryanDavis|BryanDavis]]
* 22:46 [[etherpad:p/Kay@31]] was archived as [[etherpadbackup:Kay@31]] by [[User:BryanDavis|BryanDavis]]
* 22:46 [[etherpad:p/Kafka-udp2log]] was archived as [[etherpadbackup:Kafka-udp2log]] by [[User:BryanDavis|BryanDavis]]
* 22:45 [[etherpad:p/K]] was archived as [[etherpadbackup:K]] by [[User:BryanDavis|BryanDavis]]
* 22:45 [[etherpad:p/J]] was archived as [[etherpadbackup:J]] by [[User:BryanDavis|BryanDavis]]
* 22:44 [[etherpad:p/Introduction_to_Phabricator_(WikiCon)]] was archived as [[etherpadbackup:Introduction_to_Phabricator_(WikiCon)]] by [[User:BryanDavis|BryanDavis]]
* 22:44 [[etherpad:p/Intro_a_las_tec_del_movimiento_Wikimedia]] was archived as [[etherpadbackup:Intro_a_las_tec_del_movimiento_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 22:43 [[etherpad:p/InlineScripting]] was archived as [[etherpadbackup:InlineScripting]] by [[User:BryanDavis|BryanDavis]]
* 22:43 [[etherpad:p/Incident-Replay-Sessions]] was archived as [[etherpadbackup:Incident-Replay-Sessions]] by [[User:BryanDavis|BryanDavis]]
* 22:42 [[etherpad:p/IlszcZZEEq]] was archived as [[etherpadbackup:IlszcZZEEq]] by [[User:BryanDavis|BryanDavis]]
* 22:42 [[etherpad:p/Icon_standardisation]] was archived as [[etherpadbackup:Icon_standardisation]] by [[User:BryanDavis|BryanDavis]]
* 22:41 [[etherpad:p/INGLW4jCQk]] was archived as [[etherpadbackup:INGLW4jCQk]] by [[User:BryanDavis|BryanDavis]]
* 22:41 [[etherpad:p/IBrokeWikipediaList]] was archived as [[etherpadbackup:IBrokeWikipediaList]] by [[User:BryanDavis|BryanDavis]]
* 22:40 [[etherpad:p/I]] was archived as [[etherpadbackup:I]] by [[User:BryanDavis|BryanDavis]]
* 22:39 [[etherpad:p/HuggleCon-2-planning]] was archived as [[etherpadbackup:HuggleCon-2-planning]] by [[User:BryanDavis|BryanDavis]]
* 22:39 [[etherpad:p/HuggleCon-1-participants-list]] was archived as [[etherpadbackup:HuggleCon-1-participants-list]] by [[User:BryanDavis|BryanDavis]]
* 22:38 [[etherpad:p/Huggle-dev-pad]] was archived as [[etherpadbackup:Huggle-dev-pad]] by [[User:BryanDavis|BryanDavis]]
* 22:38 [[etherpad:p/Huggle-IRC-conference]] was archived as [[etherpadbackup:Huggle-IRC-conference]] by [[User:BryanDavis|BryanDavis]]
* 22:37 [[etherpad:p/Huggle]] was archived as [[etherpadbackup:Huggle]] by [[User:BryanDavis|BryanDavis]]
* 22:37 [[etherpad:p/HowToReleaseAnApp]] was archived as [[etherpadbackup:HowToReleaseAnApp]] by [[User:BryanDavis|BryanDavis]]
* 22:36 [[etherpad:p/Hiera]] was archived as [[etherpadbackup:Hiera]] by [[User:BryanDavis|BryanDavis]]
* 22:36 [[etherpad:p/HadoopEtherpad]] was archived as [[etherpadbackup:HadoopEtherpad]] by [[User:BryanDavis|BryanDavis]]
* 22:35 [[etherpad:p/Hackathon_Showcase_&amp;_Coolest_Tool_Awards]] was archived as [[etherpadbackup:Hackathon_Showcase_&amp;_Coolest_Tool_Awards]] by [[User:BryanDavis|BryanDavis]]
* 22:35 [[etherpad:p/HKzL6x5e6z]] was archived as [[etherpadbackup:HKzL6x5e6z]] by [[User:BryanDavis|BryanDavis]]
* 22:34 [[etherpad:p/HJRCD1b81bHXm-ZS-l9w]] was archived as [[etherpadbackup:HJRCD1b81bHXm-ZS-l9w]] by [[User:BryanDavis|BryanDavis]]
* 22:34 [[etherpad:p/Gwods9UzuoEMR79NM8nl]] was archived as [[etherpadbackup:Gwods9UzuoEMR79NM8nl]] by [[User:BryanDavis|BryanDavis]]
* 22:33 [[etherpad:p/GroupSelectOption]] was archived as [[etherpadbackup:GroupSelectOption]] by [[User:BryanDavis|BryanDavis]]
* 22:33 [[etherpad:p/Google_Slack_Zoom_YouTube_bugs_in_Movement]] was archived as [[etherpadbackup:Google_Slack_Zoom_YouTube_bugs_in_Movement]] by [[User:BryanDavis|BryanDavis]]
* 22:32 [[etherpad:p/GitWorkflow]] was archived as [[etherpadbackup:GitWorkflow]] by [[User:BryanDavis|BryanDavis]]
* 22:32 [[etherpad:p/Git-migration-tools]] was archived as [[etherpadbackup:Git-migration-tools]] by [[User:BryanDavis|BryanDavis]]
* 22:31 [[etherpad:p/Git-migration]] was archived as [[etherpadbackup:Git-migration]] by [[User:BryanDavis|BryanDavis]]
* 22:31 [[etherpad:p/Git-Feb13]] was archived as [[etherpadbackup:Git-Feb13]] by [[User:BryanDavis|BryanDavis]]
* 22:30 [[etherpad:p/Gerrit_login_problems]] was archived as [[etherpadbackup:Gerrit_login_problems]] by [[User:BryanDavis|BryanDavis]]
* 22:30 [[etherpad:p/Gerrit_API_Wikimedia]] was archived as [[etherpadbackup:Gerrit_API_Wikimedia]] by [[User:BryanDavis|BryanDavis]]
* 22:29 [[etherpad:p/GerritCleanupOrg]] was archived as [[etherpadbackup:GerritCleanupOrg]] by [[User:BryanDavis|BryanDavis]]
* 22:29 [[etherpad:p/GerritAccounts]] was archived as [[etherpadbackup:GerritAccounts]] by [[User:BryanDavis|BryanDavis]]
* 22:28 [[etherpad:p/GeoData]] was archived as [[etherpadbackup:GeoData]] by [[User:BryanDavis|BryanDavis]]
* 22:28 [[etherpad:p/Gather_Q4]] was archived as [[etherpadbackup:Gather_Q4]] by [[User:BryanDavis|BryanDavis]]
* 22:27 [[etherpad:p/G__________________]] was archived as [[etherpadbackup:G__________________]] by [[User:BryanDavis|BryanDavis]]
* 22:27 [[etherpad:p/GWToolsetReview]] was archived as [[etherpadbackup:GWToolsetReview]] by [[User:BryanDavis|BryanDavis]]
* 22:26 [[etherpad:p/G]] was archived as [[etherpadbackup:G]] by [[User:BryanDavis|BryanDavis]]
* 22:26 [[etherpad:p/FutureOfEditingTalk]] was archived as [[etherpadbackup:FutureOfEditingTalk]] by [[User:BryanDavis|BryanDavis]]
* 22:25 [[etherpad:p/FrontEndDevOnboarding-bahodir]] was archived as [[etherpadbackup:FrontEndDevOnboarding-bahodir]] by [[User:BryanDavis|BryanDavis]]
* 22:25 [[etherpad:p/FrontEndDevOnboarding]] was archived as [[etherpadbackup:FrontEndDevOnboarding]] by [[User:BryanDavis|BryanDavis]]
* 22:24 [[etherpad:p/France_-_communes_-_Data_.tab]] was archived as [[etherpadbackup:France_-_communes_-_Data_.tab]] by [[User:BryanDavis|BryanDavis]]
* 22:24 [[etherpad:p/Foo]] was archived as [[etherpadbackup:Foo]] by [[User:BryanDavis|BryanDavis]]
* 22:23 [[etherpad:p/FlowTodo]] was archived as [[etherpadbackup:FlowTodo]] by [[User:BryanDavis|BryanDavis]]
* 22:23 [[etherpad:p/FlowAsAService]] was archived as [[etherpadbackup:FlowAsAService]] by [[User:BryanDavis|BryanDavis]]
* 22:22 [[etherpad:p/Flow-ops-l_mail]] was archived as [[etherpadbackup:Flow-ops-l_mail]] by [[User:BryanDavis|BryanDavis]]
* 22:21 [[etherpad:p/First_URLs]] was archived as [[etherpadbackup:First_URLs]] by [[User:BryanDavis|BryanDavis]]
* 22:21 [[etherpad:p/Feb6Outage]] was archived as [[etherpadbackup:Feb6Outage]] by [[User:BryanDavis|BryanDavis]]
* 22:20 [[etherpad:p/Feb-6-Issues]] was archived as [[etherpadbackup:Feb-6-Issues]] by [[User:BryanDavis|BryanDavis]]
* 22:20 [[etherpad:p/FeaturesTeam2012-W15]] was archived as [[etherpadbackup:FeaturesTeam2012-W15]] by [[User:BryanDavis|BryanDavis]]
* 22:19 [[etherpad:p/FeaturesTeam2012-W14]] was archived as [[etherpadbackup:FeaturesTeam2012-W14]] by [[User:BryanDavis|BryanDavis]]
* 22:19 [[etherpad:p/FeaturesTeam2012-W13]] was archived as [[etherpadbackup:FeaturesTeam2012-W13]] by [[User:BryanDavis|BryanDavis]]
* 22:18 [[etherpad:p/FeaturesTeam2012-W08]] was archived as [[etherpadbackup:FeaturesTeam2012-W08]] by [[User:BryanDavis|BryanDavis]]
* 22:18 [[etherpad:p/FeaturesTeam2012-W07]] was archived as [[etherpadbackup:FeaturesTeam2012-W07]] by [[User:BryanDavis|BryanDavis]]
* 22:17 [[etherpad:p/FeaturesTeam2012-W06]] was archived as [[etherpadbackup:FeaturesTeam2012-W06]] by [[User:BryanDavis|BryanDavis]]
* 22:17 [[etherpad:p/FeaturesTeam2012-W03]] was archived as [[etherpadbackup:FeaturesTeam2012-W03]] by [[User:BryanDavis|BryanDavis]]
* 22:16 [[etherpad:p/FeaturesTeam2012-W011]] was archived as [[etherpadbackup:FeaturesTeam2012-W011]] by [[User:BryanDavis|BryanDavis]]
* 22:16 [[etherpad:p/FeaturesTeam20111227]] was archived as [[etherpadbackup:FeaturesTeam20111227]] by [[User:BryanDavis|BryanDavis]]
* 22:15 [[etherpad:p/FeaturesTeam20111213]] was archived as [[etherpadbackup:FeaturesTeam20111213]] by [[User:BryanDavis|BryanDavis]]
* 22:15 [[etherpad:p/FeaturesTeam20111206]] was archived as [[etherpadbackup:FeaturesTeam20111206]] by [[User:BryanDavis|BryanDavis]]
* 22:14 [[etherpad:p/FeaturesTeam-Preload]] was archived as [[etherpadbackup:FeaturesTeam-Preload]] by [[User:BryanDavis|BryanDavis]]
* 22:14 [[etherpad:p/Farm101]] was archived as [[etherpadbackup:Farm101]] by [[User:BryanDavis|BryanDavis]]
* 22:13 [[etherpad:p/FRAnalyticsQ2FY16]] was archived as [[etherpadbackup:FRAnalyticsQ2FY16]] by [[User:BryanDavis|BryanDavis]]
* 22:13 [[etherpad:p/FG3gMpB1qc]] was archived as [[etherpadbackup:FG3gMpB1qc]] by [[User:BryanDavis|BryanDavis]]
* 22:12 [[etherpad:p/F2pDHw25sm7eJS1Q3Tk2]] was archived as [[etherpadbackup:F2pDHw25sm7eJS1Q3Tk2]] by [[User:BryanDavis|BryanDavis]]
* 22:12 [[etherpad:p/F2gsk6WTo1]] was archived as [[etherpadbackup:F2gsk6WTo1]] by [[User:BryanDavis|BryanDavis]]
* 22:11 [[etherpad:p/F]] was archived as [[etherpadbackup:F]] by [[User:BryanDavis|BryanDavis]]
* 22:11 [[etherpad:p/ErAnycbwZA]] was archived as [[etherpadbackup:ErAnycbwZA]] by [[User:BryanDavis|BryanDavis]]
* 22:10 [[etherpad:p/EpicIndiaItinerary]] was archived as [[etherpadbackup:EpicIndiaItinerary]] by [[User:BryanDavis|BryanDavis]]
* 22:10 [[etherpad:p/Encuentro_técnico_hispanohablante_Wikimania_2022]] was archived as [[etherpadbackup:Encuentro_técnico_hispanohablante_Wikimania_2022]] by [[User:BryanDavis|BryanDavis]]
* 22:09 [[etherpad:p/EchoNotification]] was archived as [[etherpadbackup:EchoNotification]] by [[User:BryanDavis|BryanDavis]]
* 22:09 [[etherpad:p/EQIAD-rollout-sequence]] was archived as [[etherpadbackup:EQIAD-rollout-sequence]] by [[User:BryanDavis|BryanDavis]]
* 22:08 [[etherpad:p/EL_python]] was archived as [[etherpadbackup:EL_python]] by [[User:BryanDavis|BryanDavis]]
* 22:08 [[etherpad:p/E3Analytics]] was archived as [[etherpadbackup:E3Analytics]] by [[User:BryanDavis|BryanDavis]]
* 22:07 [[etherpad:p/E3-2013-03-07-deploy]] was archived as [[etherpadbackup:E3-2013-03-07-deploy]] by [[User:BryanDavis|BryanDavis]]
* 22:07 [[etherpad:p/DrinkingShoppingList]] was archived as [[etherpadbackup:DrinkingShoppingList]] by [[User:BryanDavis|BryanDavis]]
* 22:06 [[etherpad:p/DownWithTwentyPercent]] was archived as [[etherpadbackup:DownWithTwentyPercent]] by [[User:BryanDavis|BryanDavis]]
* 22:06 [[etherpad:p/Discussion__Redefine_Wikimedia_Hackathons]] was archived as [[etherpadbackup:Discussion__Redefine_Wikimedia_Hackathons]] by [[User:BryanDavis|BryanDavis]]
* 22:05 [[etherpad:p/DevelopingMobileApps]] was archived as [[etherpadbackup:DevelopingMobileApps]] by [[User:BryanDavis|BryanDavis]]
* 22:05 [[etherpad:p/DevSummitMain]] was archived as [[etherpadbackup:DevSummitMain]] by [[User:BryanDavis|BryanDavis]]
* 22:04 [[etherpad:p/DevSum2015_designresearch]] was archived as [[etherpadbackup:DevSum2015_designresearch]] by [[User:BryanDavis|BryanDavis]]
* 22:03 [[etherpad:p/DeploymentPrepReConfig]] was archived as [[etherpadbackup:DeploymentPrepReConfig]] by [[User:BryanDavis|BryanDavis]]
* 22:03 [[etherpad:p/DSEorCDH4]] was archived as [[etherpadbackup:DSEorCDH4]] by [[User:BryanDavis|BryanDavis]]
* 22:02 [[etherpad:p/DRworkshopParkingLot]] was archived as [[etherpadbackup:DRworkshopParkingLot]] by [[User:BryanDavis|BryanDavis]]
* 22:02 [[etherpad:p/DRMethodsMenuSite]] was archived as [[etherpadbackup:DRMethodsMenuSite]] by [[User:BryanDavis|BryanDavis]]
* 22:01 [[etherpad:p/ContinuousIntegration]] was archived as [[etherpadbackup:ContinuousIntegration]] by [[User:BryanDavis|BryanDavis]]
* 22:01 [[etherpad:p/CollectionsNameA]] was archived as [[etherpadbackup:CollectionsNameA]] by [[User:BryanDavis|BryanDavis]]
* 22:00 [[etherpad:p/Code-in_student_meeting]] was archived as [[etherpadbackup:Code-in_student_meeting]] by [[User:BryanDavis|BryanDavis]]
* 22:00 [[etherpad:p/CmMGbzxbZw]] was archived as [[etherpadbackup:CmMGbzxbZw]] by [[User:BryanDavis|BryanDavis]]
* 21:59 [[etherpad:p/Chromium-Upstream]] was archived as [[etherpadbackup:Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]]
* 21:59 [[etherpad:p/CentralNoticeDiscussion]] was archived as [[etherpadbackup:CentralNoticeDiscussion]] by [[User:BryanDavis|BryanDavis]]
* 21:58 [[etherpad:p/CentralNotice]] was archived as [[etherpadbackup:CentralNotice]] by [[User:BryanDavis|BryanDavis]]
* 21:58 [[etherpad:p/CampaignAssociatedMixins]] was archived as [[etherpadbackup:CampaignAssociatedMixins]] by [[User:BryanDavis|BryanDavis]]
* 21:57 [[etherpad:p/C_________]] was archived as [[etherpadbackup:C_________]] by [[User:BryanDavis|BryanDavis]]
* 21:57 [[etherpad:p/CZg-suME0Sx-nnCfKIEW]] was archived as [[etherpadbackup:CZg-suME0Sx-nnCfKIEW]] by [[User:BryanDavis|BryanDavis]]
* 21:56 [[etherpad:p/CSF_Katie]] was archived as [[etherpadbackup:CSF_Katie]] by [[User:BryanDavis|BryanDavis]]
* 21:56 [[etherpad:p/CEE-l10n-levels]] was archived as [[etherpadbackup:CEE-l10n-levels]] by [[User:BryanDavis|BryanDavis]]
* 21:55 [[etherpad:p/C-Lessons]] was archived as [[etherpadbackup:C-Lessons]] by [[User:BryanDavis|BryanDavis]]
* 21:55 [[etherpad:p/BotLicensing]] was archived as [[etherpadbackup:BotLicensing]] by [[User:BryanDavis|BryanDavis]]
* 21:54 [[etherpad:p/BetaClusterpriorityworksync]] was archived as [[etherpadbackup:BetaClusterpriorityworksync]] by [[User:BryanDavis|BryanDavis]]
* 21:54 [[etherpad:p/ArchitectureCommittee-2015-02-04]] was archived as [[etherpadbackup:ArchitectureCommittee-2015-02-04]] by [[User:BryanDavis|BryanDavis]]
* 21:53 [[etherpad:p/ArchDoc]] was archived as [[etherpadbackup:ArchDoc]] by [[User:BryanDavis|BryanDavis]]
* 21:53 [[etherpad:p/AppsReadership]] was archived as [[etherpadbackup:AppsReadership]] by [[User:BryanDavis|BryanDavis]]
* 21:52 [[etherpad:p/AppsQ120142015]] was archived as [[etherpadbackup:AppsQ120142015]] by [[User:BryanDavis|BryanDavis]]
* 21:52 [[etherpad:p/AppsPhabelloMigration]] was archived as [[etherpadbackup:AppsPhabelloMigration]] by [[User:BryanDavis|BryanDavis]]
* 21:51 [[etherpad:p/AppsNav]] was archived as [[etherpadbackup:AppsNav]] by [[User:BryanDavis|BryanDavis]]
* 21:51 [[etherpad:p/AppsContentService]] was archived as [[etherpadbackup:AppsContentService]] by [[User:BryanDavis|BryanDavis]]
* 21:50 [[etherpad:p/AppUserTesting]] was archived as [[etherpadbackup:AppUserTesting]] by [[User:BryanDavis|BryanDavis]]
* 21:50 [[etherpad:p/App-20design-20tweaks]] was archived as [[etherpadbackup:App-20design-20tweaks]] by [[User:BryanDavis|BryanDavis]]
* 21:49 [[etherpad:p/Android_front-end_dev_task]] was archived as [[etherpadbackup:Android_front-end_dev_task]] by [[User:BryanDavis|BryanDavis]]
* 21:48 [[etherpad:p/Analytics_all_staff]] was archived as [[etherpadbackup:Analytics_all_staff]] by [[User:BryanDavis|BryanDavis]]
* 21:48 [[etherpad:p/Analytics_Wikimania]] was archived as [[etherpadbackup:Analytics_Wikimania]] by [[User:BryanDavis|BryanDavis]]
* 21:47 [[etherpad:p/AnalyticsPlanningMeetingNotes]] was archived as [[etherpadbackup:AnalyticsPlanningMeetingNotes]] by [[User:BryanDavis|BryanDavis]]
* 21:47 [[etherpad:p/AnalyticsMilestonesAndPriorities]] was archived as [[etherpadbackup:AnalyticsMilestonesAndPriorities]] by [[User:BryanDavis|BryanDavis]]
* 21:46 [[etherpad:p/AnalyticsDevStaff]] was archived as [[etherpadbackup:AnalyticsDevStaff]] by [[User:BryanDavis|BryanDavis]]
* 21:46 [[etherpad:p/Analytics-Reseach]] was archived as [[etherpadbackup:Analytics-Reseach]] by [[User:BryanDavis|BryanDavis]]
* 21:45 [[etherpad:p/Analytics-Pixel-Service]] was archived as [[etherpadbackup:Analytics-Pixel-Service]] by [[User:BryanDavis|BryanDavis]]
* 21:45 [[etherpad:p/Alpha-beta_cleaning]] was archived as [[etherpadbackup:Alpha-beta_cleaning]] by [[User:BryanDavis|BryanDavis]]
* 21:44 [[etherpad:p/Adv_Browser_Tests_-_T86593]] was archived as [[etherpadbackup:Adv_Browser_Tests_-_T86593]] by [[User:BryanDavis|BryanDavis]]
* 21:44 [[etherpad:p/Adedolapo_-_survey_for_content_Translation_users]] was archived as [[etherpadbackup:Adedolapo_-_survey_for_content_Translation_users]] by [[User:BryanDavis|BryanDavis]]
* 21:43 [[etherpad:p/Accessibility_WMDS]] was archived as [[etherpadbackup:Accessibility_WMDS]] by [[User:BryanDavis|BryanDavis]]
* 21:43 [[etherpad:p/AWMD_IRC-Meeting-13]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-13]] by [[User:BryanDavis|BryanDavis]]
* 21:42 [[etherpad:p/AWMD_IRC-Meeting-12]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-12]] by [[User:BryanDavis|BryanDavis]]
* 21:42 [[etherpad:p/AWMD_IRC-Meeting-11]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-11]] by [[User:BryanDavis|BryanDavis]]
* 21:41 [[etherpad:p/AWMD_IRC-Meeting-10]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-10]] by [[User:BryanDavis|BryanDavis]]
* 21:41 [[etherpad:p/API-blog-post]] was archived as [[etherpadbackup:API-blog-post]] by [[User:BryanDavis|BryanDavis]]
* 21:40 [[etherpad:p/AFT5LaunchTests]] was archived as [[etherpadbackup:AFT5LaunchTests]] by [[User:BryanDavis|BryanDavis]]
* 21:40 [[etherpad:p/AFT5]] was archived as [[etherpadbackup:AFT5]] by [[User:BryanDavis|BryanDavis]]
* 21:39 [[etherpad:p/9RFl8QljDZ]] was archived as [[etherpadbackup:9RFl8QljDZ]] by [[User:BryanDavis|BryanDavis]]
* 21:39 [[etherpad:p/92]] was archived as [[etherpadbackup:92]] by [[User:BryanDavis|BryanDavis]]
* 21:38 [[etherpad:p/8r8ihQ7GpX]] was archived as [[etherpadbackup:8r8ihQ7GpX]] by [[User:BryanDavis|BryanDavis]]
* 21:38 [[etherpad:p/8D1ml5Q1Sb]] was archived as [[etherpadbackup:8D1ml5Q1Sb]] by [[User:BryanDavis|BryanDavis]]
* 21:37 [[etherpad:p/88952]] was archived as [[etherpadbackup:88952]] by [[User:BryanDavis|BryanDavis]]
* 21:37 [[etherpad:p/7hDQxehZZW]] was archived as [[etherpadbackup:7hDQxehZZW]] by [[User:BryanDavis|BryanDavis]]
* 21:36 [[etherpad:p/7YAmoRjGSAvfJrj9V9sD]] was archived as [[etherpadbackup:7YAmoRjGSAvfJrj9V9sD]] by [[User:BryanDavis|BryanDavis]]
* 21:36 [[etherpad:p/7FtE43Tuhu]] was archived as [[etherpadbackup:7FtE43Tuhu]] by [[User:BryanDavis|BryanDavis]]
* 21:35 [[etherpad:p/77SMYoPXek]] was archived as [[etherpadbackup:77SMYoPXek]] by [[User:BryanDavis|BryanDavis]]
* 21:35 [[etherpad:p/6gR8aSREkz]] was archived as [[etherpadbackup:6gR8aSREkz]] by [[User:BryanDavis|BryanDavis]]
* 21:34 [[etherpad:p/58040]] was archived as [[etherpadbackup:58040]] by [[User:BryanDavis|BryanDavis]]
* 21:34 [[etherpad:p/503-eqiad-cirrussearch]] was archived as [[etherpadbackup:503-eqiad-cirrussearch]] by [[User:BryanDavis|BryanDavis]]
* 21:33 [[etherpad:p/4pKxno7DAI]] was archived as [[etherpadbackup:4pKxno7DAI]] by [[User:BryanDavis|BryanDavis]]
* 21:33 [[etherpad:p/4J8fHuL0yB]] was archived as [[etherpadbackup:4J8fHuL0yB]] by [[User:BryanDavis|BryanDavis]]
* 21:32 [[etherpad:p/4.1_launch_post-mortem]] was archived as [[etherpadbackup:4.1_launch_post-mortem]] by [[User:BryanDavis|BryanDavis]]
* 21:32 [[etherpad:p/20percent]] was archived as [[etherpadbackup:20percent]] by [[User:BryanDavis|BryanDavis]]
* 21:31 [[etherpad:p/2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] was archived as [[etherpadbackup:2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] by [[User:BryanDavis|BryanDavis]]
* 21:30 [[etherpad:p/2025-10-workers-reboot]] was archived as [[etherpadbackup:2025-10-workers-reboot]] by [[User:BryanDavis|BryanDavis]]
* 21:30 [[etherpad:p/2025-04_toolforge_monthly_update]] was archived as [[etherpadbackup:2025-04_toolforge_monthly_update]] by [[User:BryanDavis|BryanDavis]]
* 21:29 [[etherpad:p/2024_tekton_upgrade]] was archived as [[etherpadbackup:2024_tekton_upgrade]] by [[User:BryanDavis|BryanDavis]]
* 21:29 [[etherpad:p/2021-switchdc-testing]] was archived as [[etherpadbackup:2021-switchdc-testing]] by [[User:BryanDavis|BryanDavis]]
* 21:28 [[etherpad:p/2021-switchdc-prep]] was archived as [[etherpadbackup:2021-switchdc-prep]] by [[User:BryanDavis|BryanDavis]]
* 21:28 [[etherpad:p/20205-06-components-api-reviews]] was archived as [[etherpadbackup:20205-06-components-api-reviews]] by [[User:BryanDavis|BryanDavis]]
* 21:27 [[etherpad:p/2019-09-25-cloudvirt-load-rebalance]] was archived as [[etherpadbackup:2019-09-25-cloudvirt-load-rebalance]] by [[User:BryanDavis|BryanDavis]]
* 21:27 [[etherpad:p/2019-07-23-eqiad-asw-a]] was archived as [[etherpadbackup:2019-07-23-eqiad-asw-a]] by [[User:BryanDavis|BryanDavis]]
* 21:26 [[etherpad:p/2019-07-16-docker-registry]] was archived as [[etherpadbackup:2019-07-16-docker-registry]] by [[User:BryanDavis|BryanDavis]]
* 21:26 [[etherpad:p/20180829-labstore-reboots]] was archived as [[etherpadbackup:20180829-labstore-reboots]] by [[User:BryanDavis|BryanDavis]]
* 21:25 [[etherpad:p/20150519-Mailman]] was archived as [[etherpadbackup:20150519-Mailman]] by [[User:BryanDavis|BryanDavis]]
* 21:25 [[etherpad:p/2014-antoine-june-checkin]] was archived as [[etherpadbackup:2014-antoine-june-checkin]] by [[User:BryanDavis|BryanDavis]]
* 21:24 [[etherpad:p/2013-2014-Q2-MobileWeb]] was archived as [[etherpadbackup:2013-2014-Q2-MobileWeb]] by [[User:BryanDavis|BryanDavis]]
* 21:24 [[etherpad:p/2012-2013-Q1-Mobile-Dept]] was archived as [[etherpadbackup:2012-2013-Q1-Mobile-Dept]] by [[User:BryanDavis|BryanDavis]]
* 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]]
* 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]]
* 20:22 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]

=== 2026-05-03 ===
* 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 13:56 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]]
* 13:14 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]]
* 13:11 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/WIKISOO3]] was archived as [[etherpadbackup:WIKISOO3]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/VertalingWikiIt]] was archived as [[etherpadbackup:VertalingWikiIt]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]]
* 12:02 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]]
* 12:00 [[etherpad:p/BugTriage-Collection]] was archived as [[etherpadbackup:BugTriage-Collection]] by [[User:BryanDavis|BryanDavis]]
* 09:25 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 09:18 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 07:46 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 07:28 [[etherpad:p/prodlikevagrant-hackathon-planning]] was archived as [[etherpadbackup:prodlikevagrant-hackathon-planning]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-05-02 ===
* 13:01 [[etherpad:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] was archived as [[etherpadbackup:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 07:18 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 06:58 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C&gt;%2C&lt;%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-05-01 ===
* 18:12 [[etherpad:p/10myths]] was archived as [[etherpadbackup:p/10myths]] by [[User:Krinkle|Krinkle]]
* 15:30 [[etherpad:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] was archived as [[etherpadbackup:p/test-*,?,\,/,%,&gt;,&lt;,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 14:50 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:Jon Harald Søby|Jon Harald Søby]]
* 14:42 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 13:51 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 13:16 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 13:07 [[etherpad:p/copyvios-todo]] was archived as [[etherpadbackup:p/copyvios-todo]] by [[User:Chlod|Chlod]]
* 10:05 [[etherpad:p/hhvmdeploy]] was archived as [[etherpadbackup:p/hhvmdeploy]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-04-29 ===
* 00:49 [[etherpad:p/better-security-patch-management]] was archived as [[etherpadbackup:p/better-security-patch-management]] by [[User:BDavis (WMF)|BDavis (WMF)]]
* 00:49 [[etherpad:p/rooks-questions-to-alex]] was archived as [[etherpadbackup:p/rooks-questions-to-alex]] by [[User:BDavis (WMF)|BDavis (WMF)]]

=== 2026-04-27 ===
* 04:44 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 03:46 [[etherpad:p/wmh2023-Overview_of_technical_areas_&amp;_projects]] was archived as [[etherpadbackup:p/wmh2023-Overview_of_technical_areas_&amp;_projects]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/open_meeting_COVID-19_18/04]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_18/04]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/open_meeting_COVID-19_04/05]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_04/05]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/aswikimeet/july-2024]] was archived as [[etherpadbackup:p/aswikimeet/july-2024]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/Wikisource_Meetup_29/09/2020]] was archived as [[etherpadbackup:p/Wikisource_Meetup_29/09/2020]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] was archived as [[etherpadbackup:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/Wikimedia_LGBT+/Governance/2023-10-13]] was archived as [[etherpadbackup:p/Wikimedia_LGBT+/Governance/2023-10-13]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopD]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopD]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopC]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopC]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopB]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopB]] by [[User:BryanDavis|BryanDavis]]
* 03:45 [[etherpad:p/WikiCendekia2026/WarkopA]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopA]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Stolpersteine_nach_WikiData_(01/2025]] was archived as [[etherpadbackup:p/Stolpersteine_nach_WikiData_(01/2025]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Sommer_24/8,_1._August]] was archived as [[etherpadbackup:p/Sommer_24/8,_1._August]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] was archived as [[etherpadbackup:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Meetup/Cape_Town/Cape_Town_22]] was archived as [[etherpadbackup:p/Meetup/Cape_Town/Cape_Town_22]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] was archived as [[etherpadbackup:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] was archived as [[etherpadbackup:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] was archived as [[etherpadbackup:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_C]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_C]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_B]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_B]] by [[User:BryanDavis|BryanDavis]]
* 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_A]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_A]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] was archived as [[etherpadbackup:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] was archived as [[etherpadbackup:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Wikispore]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Wikispore]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Translatable_modules]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Translatable_modules]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Movement_Strategy]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Movement_Strategy]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Lua_templates_for_beginners]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Lua_templates_for_beginners]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/Help_your_community_grow]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Help_your_community_grow]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/CEE_Meeting_2020/101_Ways_to_Contribute]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/101_Ways_to_Contribute]] by [[User:BryanDavis|BryanDavis]]
* 03:40 [[etherpad:p/AsWikiMentoringProject2024/Oct-Dec]] was archived as [[etherpadbackup:p/AsWikiMentoringProject2024/Oct-Dec]] by [[User:BryanDavis|BryanDavis]]
* 03:16 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]]
* 03:15 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:wmh2023-Overview_of_technical_areas_%26_projects|wmh2023-Overview_of_technical_areas_%26_projects]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_20%2F04|open_meeting_COVID-19_20%2F04]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_18%2F04|open_meeting_COVID-19_18%2F04]]
* 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_04%2F05|open_meeting_COVID-19_04%2F05]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:aswikimeet%2Fjuly-2024|aswikimeet%2Fjuly-2024]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikisource_Meetup_29%2F09%2F2020|Wikisource_Meetup_29%2F09%2F2020]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020|Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedia_LGBT+%2FGovernance%2F2023-10-13|Wikimedia_LGBT+%2FGovernance%2F2023-10-13]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopD|WikiCendekia2026%2FWarkopD]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopC|WikiCendekia2026%2FWarkopC]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopB|WikiCendekia2026%2FWarkopB]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopA|WikiCendekia2026%2FWarkopA]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Summary%2F_Hypothesis|Summary%2F_Hypothesis]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(12%2F2024)|Stolpersteine_nach_WikiData_(12%2F2024)]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(01%2F2025|Stolpersteine_nach_WikiData_(01%2F2025]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Sommer_24%2F8,_1._August|Sommer_24%2F8,_1._August]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala|SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Meetup%2FCape_Town%2FCape_Town_22|Meetup%2FCape_Town%2FCape_Town_22]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk|Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23|Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Igbo_Wikimedians_Community%2FMovement_Charter_Commun|Igbo_Wikimedians_Community%2FMovement_Charter_Commun]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_C|GLAM-Treffen_2020%2FGruppe_C]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_B|GLAM-Treffen_2020%2FGruppe_B]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_A|GLAM-Treffen_2020%2FGruppe_A]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Commons_Photographers_30%2F05%2F2020|Commons_Photographers_30%2F05%2F2020]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:ChristianSW%2FNotizen|ChristianSW%2FNotizen]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202|CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FOutlook_CEE_Meeting_2022_V2|CEE_Talks%2FOutlook_CEE_Meeting_2022_V2]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FWikispore|CEE_Meeting_2020%2FWikispore]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FTranslatable_modules|CEE_Meeting_2020%2FTranslatable_modules]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref|CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FMovement_Strategy|CEE_Meeting_2020%2FMovement_Strategy]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FLua_templates_for_beginners|CEE_Meeting_2020%2FLua_templates_for_beginners]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FHelp_your_community_grow|CEE_Meeting_2020%2FHelp_your_community_grow]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2F101_Ways_to_Contribute|CEE_Meeting_2020%2F101_Ways_to_Contribute]]
* 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:AsWikiMentoringProject2024%2FOct-Dec|AsWikiMentoringProject2024%2FOct-Dec]]
* 01:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:51 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:49 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:43 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]]
* 00:05 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]

=== 2026-04-26 ===
* 23:25 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 21:58 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 17:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 17:22 [[etherpad:p/bd808 test/path/?query=param]] was archived as [[etherpadbackup:p/bd808 test/path/?query=param]] by [[User:BryanDavis|BryanDavis]]
* 17:22 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 16:17 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]

=== 2026-04-20 ===
* 02:42 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]]
* 02:36 [[etherpad:p/bd808-etherpad-backup-test]] was archived by [[User:BryanDavis]]</text>
      <sha1>53a0u2zul9j4y6cxr6ynes3igynur60</sha1>
    </revision>
  </page>
  <page>
    <title>User:Effie Mouzeli (WMF)/SCROLL/styles.css</title>
    <ns>2</ns>
    <id>460101</id>
    <revision>
      <id>2435517</id>
      <parentid>2423296</parentid>
      <timestamp>2026-07-12T01:17:21Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>flip colors back for darkmode fix</comment>
      <origin>2435517</origin>
      <model>sanitized-css</model>
      <format>text/css</format>
      <text bytes="1906" sha1="q460p1k5fb7e8ej43k49w2uieffk5yd" xml:space="preserve">.scroll-placeholder {
    color: var(--color-disabled);
    font-style: italic;
}
/*
   Green header strip at the top of the two info cards.
   Usage: add class=&quot;scroll-card-header&quot; to the &lt;th&gt; cell.
*/
.scroll-card-header {
    background: var(--background-color-success-subtle);
    font-size: 110%;
    padding: 6px;
}
/* Card label cells
   The grey left-hand label cells in the info cards.
   Usage: add class=&quot;scroll-card-label&quot; to the &lt;td&gt; cell.
*/
.scroll-card-label {
    background: var(--background-color-notice-subtle);
    width: 40%;
}
/* Section banner rows
   The pink divider rows inside checklist tables (Documentation,
   Contacting the team, etc.).
   Usage: add class=&quot;scroll-section&quot; to the &lt;td&gt; cell.
*/
.scroll-section {
    background: var(--background-color-error-subtle--hover);
    font-weight: bold;
    text-align: center;
    padding: 5px 8px;
}
/* Checklist table headers
   The sticky green column header row.
   Usage: add class=&quot;scroll-col-header&quot; to each &lt;th&gt; cell.
*/
.scroll-col-header {
    position: sticky;
    top: 0;
    z-index: 1;
    background: var(--background-color-success-subtle);
}
.scroll-col-header-center {
    position: sticky;
    top: 0;
    z-index: 1;
    background: var(--background-color-success-subtle);
    text-align: center;
}
/* Legend
   The emoji key shown between the info cards and the checklist tables.
*/
.scroll-legend {
    font-size: 85%;
    color: var(--color-subtle);
    margin: 0.8em 0 1.2em 0;
    border: none;
    width: 100%;
}
/* Remarks column content
   Grey helper text in the Remarks cells.
   Usage: &lt;span class=&quot;scroll-remarks&quot;&gt;...&lt;/span&gt;
*/
.scroll-remarks {
    color: var(--color-placeholder);
    font-size: 92%;
}
/* Assignee column content
   Grey text in the Assignee cells.
   Usage: &lt;span class=&quot;scroll-assignee&quot;&gt;...&lt;/span&gt;
*/
.scroll-assignee {
    color: var(--color-placeholder);
    font-size: 92%;
}</text>
      <sha1>q460p1k5fb7e8ej43k49w2uieffk5yd</sha1>
    </revision>
  </page>
  <page>
    <title>SCROLL/Duck</title>
    <ns>0</ns>
    <id>460178</id>
    <revision>
      <id>2435516</id>
      <parentid>2431802</parentid>
      <timestamp>2026-07-12T01:16:09Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>rm displaytitle</comment>
      <origin>2435516</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="29324" sha1="dm98h10lstrnfip401e7aw7jnnfgjsa" xml:space="preserve">{{SCROLL/Tools/Banner|completed}}
&lt;templatestyles src=&quot;User:Effie_Mouzeli_(WMF)/SCROLL/styles.css&quot; /&gt;

''Service Checklist for Readiness, Operations, Launch and Lifecycle''

__TOC__
{{DISPLAYTITLE:SCROLL: {{SUBPAGENAME}}}} 
{| style=&quot;width: 100%; border: none; background: transparent;&quot;
| style=&quot;vertical-align: top; width: 38%; padding-right: 2%;&quot; |
{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%; border: 1px solid #a2a9b1;&quot;
|-
! colspan=&quot;2&quot; class=&quot;scroll-card-header&quot; | 📜 Service/Component identity
|-
| class=&quot;scroll-card-label&quot; | '''Name'''              || Duck
|-
| class=&quot;scroll-card-label&quot; | '''Type of Request'''      || ⚙️ Service
|-
| class=&quot;scroll-card-label&quot; | '''Development Owner'''                || [https://wikitech.wikimedia.org/wiki/Duck Wikifauna Team] #wikifauna
|-
| class=&quot;scroll-card-label&quot; | '''SCROLL bearer'''        || @jolene.sre
|-
| class=&quot;scroll-card-label&quot; | '''Soft Launch Target''' (some users)   || 2026-02-15
|-
| class=&quot;scroll-card-label&quot; | '''Full Launch Target''' (all users) || 2026-04-01
|-
| class=&quot;scroll-card-label&quot; | '''SCROLL epic'''          || {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T426847}}
|}

| style=&quot;vertical-align: top; width: 60%;&quot; |
{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%; border: 1px solid #a2a9b1;&quot;
|-
! colspan=&quot;2&quot; class=&quot;scroll-card-header&quot; | Quick references
|-
| class=&quot;scroll-card-label&quot; | '''Phabricator tags'''                   || #wikifauna, #duck-service
|-
| class=&quot;scroll-card-label&quot; | '''Wikitech Page URL'''                  || [https://wikitech.wikimedia.org/wiki/Duck Duck]
|-
| class=&quot;scroll-card-label&quot; | '''Google Drive URL'''                   || [https://drive.google.com/drive/folders/duck-wikifauna Wikifauna/Duck]
|-
| class=&quot;scroll-card-label&quot; | '''Design Document'''                    || [https://docs.google.com/document/d/duck-design Duck: design overview (google doc)]
|}
|}

{| class=&quot;scroll-legend&quot;
|-
| '''Status:''' &amp;nbsp; 🟢 Done &amp;nbsp;·&amp;nbsp; 🔴 Not Done &amp;nbsp;·&amp;nbsp; '''N/A'''
|-
| '''Priority:''' &amp;nbsp; 🚀 Required for soft launch &amp;nbsp;·&amp;nbsp; 💯 Required for full launch &amp;nbsp;·&amp;nbsp; ❓ Needs scoping / may not be applicable
|-
| '''Required for:''' &amp;nbsp; ⚙️ Service &amp;nbsp;·&amp;nbsp; 🧩 Extension &amp;nbsp;·&amp;nbsp; 🌻 Core Feature &amp;nbsp;·&amp;nbsp; ✨ Feature
|}

== Service Summary ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Documentation
|-
| style=&quot;text-align: center;&quot; | '''1.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have a Wikitech page? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401012}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;https://wikitech.wikimedia.org/wiki/Duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-28
|-
| style=&quot;text-align: center;&quot; | '''1.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Is your service or component present on the [[mw:Wikimedia_Production/Service_Catalog|Service Catalogue]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T327319}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes (link to catalogue item)&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-03
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Contacting The Team
|-
| style=&quot;text-align: center;&quot; | '''1.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are the team's Phabricator tags and contact details documented on the Wikitech page and up to date in officewiki, so the right people can be found promptly when needed? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401015}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-30
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | External Reviews
|-
| style=&quot;text-align: center;&quot; | '''1.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by SRE? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401002}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;yes &lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-15
|-
| style=&quot;text-align: center;&quot; | '''1.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by Security? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401021}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, see phab task for Security related work&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @nina.sec || style=&quot;text-align: center;&quot; | 2026-02-04
|-
| style=&quot;text-align: center;&quot; | '''1.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|na}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by Data Persistence? || style=&quot;text-align: center;&quot; | ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Scoped out during Prologue: Duck service is stateless&lt;/span&gt; || style=&quot;text-align: center;&quot; |  || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-16
|}

== Operating Procedures ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Interactions
|-
| style=&quot;text-align: center;&quot; | '''2.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have you described how the service/feature interacts with common mediawiki userflows on wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401031}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-26
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Infrastructure
|-
| style=&quot;text-align: center;&quot; | '''2.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|na}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service runs on baremetal, are the server prefixes documented on wikitech? || || scoped out|| style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-16
|-
| style=&quot;text-align: center;&quot; | '''2.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service runs on Kubernetes, has it been added to [[Kubernetes/Deployment Charts|deployment charts]] (helm charts) and has the relevant [[Kubernetes/Service ports|kubernetes service defined]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401034}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401035}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401036}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Helm chart in deployment-charts/charts/duck and k8s Service of type ClusterIP, fronted by the standard ingress for public endpoints.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-05
|-
| style=&quot;text-align: center;&quot; | '''2.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If you have a Service URL? should it be present in service.yaml on Puppet? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401033}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401037}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401043}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Documented on wikitech&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@jolene.sre&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-06
|-
| style=&quot;text-align: center;&quot; | '''2.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  Does your service have a staging environment? if yes, is it documented on Wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401038}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@jolene.sre&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-07
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Service Traffic
|-
| style=&quot;text-align: center;&quot; | '''2.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Do you have an estimation of the traffic your service will be serving? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401041}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Projected ~800 req/s  across both DCs, with peaks of ~2k req/s during meme spikes. Methodology explained on phab task&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-29
|-
| style=&quot;text-align: center;&quot; | '''2.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Does your service accept traffic directly from the CDN? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401042}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes: /duck.php is exposed via the edge.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@alarik.sre&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-12
|-
| style=&quot;text-align: center;&quot; | '''2.8''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service is in discovery, can the service be depooled safely and run from a single DC, any special consideration for [[Switch Datacenter|DC Switchover]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401044}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;TODO: wikitech section&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-12
|-
| style=&quot;text-align: center;&quot; | '''2.9''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Have you documented with which systems/datastores your service needs to communicate with? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401045}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;TODO: wikitech link&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-30
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Maintenance
|-
| style=&quot;text-align: center;&quot; | '''2.10''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|na}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  If you have dependencies on maintenance scripts (mw-script) or crons (mw-cron), have they been documented and recently tested? || style=&quot;text-align: center;&quot; | ||  &lt;span class=&quot;scroll-remarks&quot;&gt;N/A: Duck has no mw-script or mw-cron &lt;/span&gt; || style=&quot;text-align: center;&quot; |  || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-16
|}

== Release Confidence ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Building and Testing
|-
| style=&quot;text-align: center;&quot; | '''3.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does your wikitech page link to the code repository and production branch, and recent image (if applicable)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401051}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401052}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-04
|-
| style=&quot;text-align: center;&quot; | '''3.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have integration and unit tests (CI)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401053}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, we use GitLab CI&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-04
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Initial Rollout
|-
| style=&quot;text-align: center;&quot; | '''3.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have a rollout plan (soft and full launch), including the common failure points you have identified for launch day? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401054}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401055}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, see phab task:&lt;ul&gt;&lt;li&gt;Rollout: per-wiki rollout&lt;/li&gt;&lt;li&gt;Launch-day failure points: cold caches at the CDN edge&lt;/li&gt;&lt;/ul&gt;&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-11
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Deploying to Production
|-
| style=&quot;text-align: center;&quot; | '''3.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have Configuration &amp; Deployment Owners? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401056}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;@bjorn.duck and @inga.duck can deploy &lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-12
|}

== Observability ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Metrics, Instrumentation, Logging
|-
| style=&quot;text-align: center;&quot; | '''4.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are you exporting Prometheus metrics and sending logs to Logstash? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401061}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, via shared-libraries&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-02
|-
| style=&quot;text-align: center;&quot; | '''4.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are key user flows and business metrics instrumented and exported? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401062}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Counters for users &quot;Like&quot; and &quot;Dislike&quot; clicks, visualised in grafana&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-14
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Service Level Objectives
|-
| style=&quot;text-align: center;&quot; | '''4.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have SLOs been agreed on and published to assist in evaluating the impact on end users? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401063}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes see [[SLO/Duck]].&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @lukas.slo || style=&quot;text-align: center;&quot; | 2026-02-20
|-
| style=&quot;text-align: center;&quot; | '''4.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have the SLO dashboard and alerts been configured in Sloth? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401064}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes see [[SLO/Duck]].&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @lukas.slo || style=&quot;text-align: center;&quot; | 2026-02-22
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Monitoring
|-
| style=&quot;text-align: center;&quot; | '''4.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are your dashboards (Grafana, Logstash) documented on  Wikitech page, and include links to related dashboards and runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401071}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401072}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401073}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes. TODO: wikitech page link&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-05
|-
| style=&quot;text-align: center;&quot; | '''4.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you monitor latency variations at the p50, p75, and p99 percentiles (eg via envoy, or other business metrics), as well as the status of external dependencies? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401074}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401075}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, latency p50/p75/p99 via Envoy, and external dependency status (memcached health, ducklings.org) is monitored.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-10
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Alerting
|-
| style=&quot;text-align: center;&quot; | '''4.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are your alerts documented on Wikitech and committed to [[Alertmanager]]: paging alerts, alerts for excessive errors (business, infrastructure, budget burn), and linked to runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401076}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401077}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401078}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-16
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Incident Response
|-
| style=&quot;text-align: center;&quot; | '''4.8''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||   Does your component have and Incident Response owner and Runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401079}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, see [[Duck#Incident response]]&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-25
|}

== Reliability and performance ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Resources
|-
| style=&quot;text-align: center;&quot; | '''5.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️ ||  Do you have an estimation of the resources you will need? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401081}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, see phabricator task&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-15
|-
| style=&quot;text-align: center;&quot; | '''5.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️ ||  Is the service designed to scale up or down as needed? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401082}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, no need to ping the devs&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-20
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Reliability
|-
| style=&quot;text-align: center;&quot; | '''5.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Does each component have its own health or liveness check to ensure production traffic does not reach an unhealthy endpoint?
| style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401083}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-08
|-
| style=&quot;text-align: center;&quot; | '''5.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Have you identified your system SPOFs? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401084}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, none &lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-19
|-
| style=&quot;text-align: center;&quot; | '''5.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does the component protect itself from being overwhelmed by incoming traffic (e.g. rate limiting, load shedding)? If yes, have they been documented? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401085}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-22
|-
| style=&quot;text-align: center;&quot; | '''5.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does the component degrade gracefully when its dependencies fail (e.g. timeouts, backoff/retry, fallback/failover, cached or default responses)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401086}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, see wikitech&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-25
|-
| style=&quot;text-align: center;&quot; | '''5.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Is the Bus Factor for this service or feature at least 2? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401087}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Three Wikifauna engineers (@bjorn.duck, @inga.duck, @oskar.duck) can deploy and operate Duck end-to-end.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-28
|}</text>
      <sha1>dm98h10lstrnfip401e7aw7jnnfgjsa</sha1>
    </revision>
    <revision>
      <id>2435518</id>
      <parentid>2435516</parentid>
      <timestamp>2026-07-12T01:18:03Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>rm 2nd displaytitle</comment>
      <origin>2435518</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="29282" sha1="or61g6973vg5726sjs1xncsw4gt2l3f" xml:space="preserve">{{SCROLL/Tools/Banner|completed}}
&lt;templatestyles src=&quot;User:Effie_Mouzeli_(WMF)/SCROLL/styles.css&quot; /&gt;

''Service Checklist for Readiness, Operations, Launch and Lifecycle''

__TOC__
{| style=&quot;width: 100%; border: none; background: transparent;&quot;
| style=&quot;vertical-align: top; width: 38%; padding-right: 2%;&quot; |
{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%; border: 1px solid #a2a9b1;&quot;
|-
! colspan=&quot;2&quot; class=&quot;scroll-card-header&quot; | 📜 Service/Component identity
|-
| class=&quot;scroll-card-label&quot; | '''Name'''              || Duck
|-
| class=&quot;scroll-card-label&quot; | '''Type of Request'''      || ⚙️ Service
|-
| class=&quot;scroll-card-label&quot; | '''Development Owner'''                || [https://wikitech.wikimedia.org/wiki/Duck Wikifauna Team] #wikifauna
|-
| class=&quot;scroll-card-label&quot; | '''SCROLL bearer'''        || @jolene.sre
|-
| class=&quot;scroll-card-label&quot; | '''Soft Launch Target''' (some users)   || 2026-02-15
|-
| class=&quot;scroll-card-label&quot; | '''Full Launch Target''' (all users) || 2026-04-01
|-
| class=&quot;scroll-card-label&quot; | '''SCROLL epic'''          || {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T426847}}
|}

| style=&quot;vertical-align: top; width: 60%;&quot; |
{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%; border: 1px solid #a2a9b1;&quot;
|-
! colspan=&quot;2&quot; class=&quot;scroll-card-header&quot; | Quick references
|-
| class=&quot;scroll-card-label&quot; | '''Phabricator tags'''                   || #wikifauna, #duck-service
|-
| class=&quot;scroll-card-label&quot; | '''Wikitech Page URL'''                  || [https://wikitech.wikimedia.org/wiki/Duck Duck]
|-
| class=&quot;scroll-card-label&quot; | '''Google Drive URL'''                   || [https://drive.google.com/drive/folders/duck-wikifauna Wikifauna/Duck]
|-
| class=&quot;scroll-card-label&quot; | '''Design Document'''                    || [https://docs.google.com/document/d/duck-design Duck: design overview (google doc)]
|}
|}

{| class=&quot;scroll-legend&quot;
|-
| '''Status:''' &amp;nbsp; 🟢 Done &amp;nbsp;·&amp;nbsp; 🔴 Not Done &amp;nbsp;·&amp;nbsp; '''N/A'''
|-
| '''Priority:''' &amp;nbsp; 🚀 Required for soft launch &amp;nbsp;·&amp;nbsp; 💯 Required for full launch &amp;nbsp;·&amp;nbsp; ❓ Needs scoping / may not be applicable
|-
| '''Required for:''' &amp;nbsp; ⚙️ Service &amp;nbsp;·&amp;nbsp; 🧩 Extension &amp;nbsp;·&amp;nbsp; 🌻 Core Feature &amp;nbsp;·&amp;nbsp; ✨ Feature
|}

== Service Summary ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Documentation
|-
| style=&quot;text-align: center;&quot; | '''1.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have a Wikitech page? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401012}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;https://wikitech.wikimedia.org/wiki/Duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-28
|-
| style=&quot;text-align: center;&quot; | '''1.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Is your service or component present on the [[mw:Wikimedia_Production/Service_Catalog|Service Catalogue]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T327319}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes (link to catalogue item)&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-03
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Contacting The Team
|-
| style=&quot;text-align: center;&quot; | '''1.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are the team's Phabricator tags and contact details documented on the Wikitech page and up to date in officewiki, so the right people can be found promptly when needed? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401015}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-30
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | External Reviews
|-
| style=&quot;text-align: center;&quot; | '''1.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by SRE? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401002}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;yes &lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-15
|-
| style=&quot;text-align: center;&quot; | '''1.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by Security? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401021}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, see phab task for Security related work&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @nina.sec || style=&quot;text-align: center;&quot; | 2026-02-04
|-
| style=&quot;text-align: center;&quot; | '''1.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|na}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by Data Persistence? || style=&quot;text-align: center;&quot; | ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Scoped out during Prologue: Duck service is stateless&lt;/span&gt; || style=&quot;text-align: center;&quot; |  || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-16
|}

== Operating Procedures ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Interactions
|-
| style=&quot;text-align: center;&quot; | '''2.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have you described how the service/feature interacts with common mediawiki userflows on wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401031}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-26
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Infrastructure
|-
| style=&quot;text-align: center;&quot; | '''2.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|na}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service runs on baremetal, are the server prefixes documented on wikitech? || || scoped out|| style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-16
|-
| style=&quot;text-align: center;&quot; | '''2.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service runs on Kubernetes, has it been added to [[Kubernetes/Deployment Charts|deployment charts]] (helm charts) and has the relevant [[Kubernetes/Service ports|kubernetes service defined]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401034}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401035}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401036}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Helm chart in deployment-charts/charts/duck and k8s Service of type ClusterIP, fronted by the standard ingress for public endpoints.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-05
|-
| style=&quot;text-align: center;&quot; | '''2.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If you have a Service URL? should it be present in service.yaml on Puppet? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401033}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401037}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401043}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Documented on wikitech&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@jolene.sre&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-06
|-
| style=&quot;text-align: center;&quot; | '''2.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  Does your service have a staging environment? if yes, is it documented on Wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401038}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@jolene.sre&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-07
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Service Traffic
|-
| style=&quot;text-align: center;&quot; | '''2.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Do you have an estimation of the traffic your service will be serving? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401041}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Projected ~800 req/s  across both DCs, with peaks of ~2k req/s during meme spikes. Methodology explained on phab task&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-29
|-
| style=&quot;text-align: center;&quot; | '''2.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Does your service accept traffic directly from the CDN? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401042}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes: /duck.php is exposed via the edge.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@alarik.sre&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-12
|-
| style=&quot;text-align: center;&quot; | '''2.8''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service is in discovery, can the service be depooled safely and run from a single DC, any special consideration for [[Switch Datacenter|DC Switchover]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401044}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;TODO: wikitech section&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-12
|-
| style=&quot;text-align: center;&quot; | '''2.9''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Have you documented with which systems/datastores your service needs to communicate with? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401045}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;TODO: wikitech link&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-30
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Maintenance
|-
| style=&quot;text-align: center;&quot; | '''2.10''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|na}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  If you have dependencies on maintenance scripts (mw-script) or crons (mw-cron), have they been documented and recently tested? || style=&quot;text-align: center;&quot; | ||  &lt;span class=&quot;scroll-remarks&quot;&gt;N/A: Duck has no mw-script or mw-cron &lt;/span&gt; || style=&quot;text-align: center;&quot; |  || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-01-16
|}

== Release Confidence ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Building and Testing
|-
| style=&quot;text-align: center;&quot; | '''3.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does your wikitech page link to the code repository and production branch, and recent image (if applicable)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401051}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401052}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-04
|-
| style=&quot;text-align: center;&quot; | '''3.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have integration and unit tests (CI)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401053}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, we use GitLab CI&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-04
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Initial Rollout
|-
| style=&quot;text-align: center;&quot; | '''3.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have a rollout plan (soft and full launch), including the common failure points you have identified for launch day? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401054}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401055}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, see phab task:&lt;ul&gt;&lt;li&gt;Rollout: per-wiki rollout&lt;/li&gt;&lt;li&gt;Launch-day failure points: cold caches at the CDN edge&lt;/li&gt;&lt;/ul&gt;&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-11
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Deploying to Production
|-
| style=&quot;text-align: center;&quot; | '''3.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have Configuration &amp; Deployment Owners? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401056}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;@bjorn.duck and @inga.duck can deploy &lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-12
|}

== Observability ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Metrics, Instrumentation, Logging
|-
| style=&quot;text-align: center;&quot; | '''4.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are you exporting Prometheus metrics and sending logs to Logstash? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401061}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, via shared-libraries&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-02
|-
| style=&quot;text-align: center;&quot; | '''4.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are key user flows and business metrics instrumented and exported? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401062}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Counters for users &quot;Like&quot; and &quot;Dislike&quot; clicks, visualised in grafana&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-14
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Service Level Objectives
|-
| style=&quot;text-align: center;&quot; | '''4.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have SLOs been agreed on and published to assist in evaluating the impact on end users? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401063}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes see [[SLO/Duck]].&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @lukas.slo || style=&quot;text-align: center;&quot; | 2026-02-20
|-
| style=&quot;text-align: center;&quot; | '''4.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have the SLO dashboard and alerts been configured in Sloth? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401064}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes see [[SLO/Duck]].&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @lukas.slo || style=&quot;text-align: center;&quot; | 2026-02-22
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Monitoring
|-
| style=&quot;text-align: center;&quot; | '''4.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are your dashboards (Grafana, Logstash) documented on  Wikitech page, and include links to related dashboards and runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401071}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401072}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401073}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes. TODO: wikitech page link&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-05
|-
| style=&quot;text-align: center;&quot; | '''4.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you monitor latency variations at the p50, p75, and p99 percentiles (eg via envoy, or other business metrics), as well as the status of external dependencies? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401074}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401075}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, latency p50/p75/p99 via Envoy, and external dependency status (memcached health, ducklings.org) is monitored.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-10
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Alerting
|-
| style=&quot;text-align: center;&quot; | '''4.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are your alerts documented on Wikitech and committed to [[Alertmanager]]: paging alerts, alerts for excessive errors (business, infrastructure, budget burn), and linked to runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401076}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401077}}, {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401078}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-16
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Incident Response
|-
| style=&quot;text-align: center;&quot; | '''4.8''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||   Does your component have and Incident Response owner and Runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401079}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes, see [[Duck#Incident response]]&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-25
|}

== Reliability and performance ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Resources
|-
| style=&quot;text-align: center;&quot; | '''5.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️ ||  Do you have an estimation of the resources you will need? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401081}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, see phabricator task&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-15
|-
| style=&quot;text-align: center;&quot; | '''5.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️ ||  Is the service designed to scale up or down as needed? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401082}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, no need to ping the devs&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-20
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Reliability
|-
| style=&quot;text-align: center;&quot; | '''5.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Does each component have its own health or liveness check to ensure production traffic does not reach an unhealthy endpoint?
| style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401083}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-08
|-
| style=&quot;text-align: center;&quot; | '''5.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Have you identified your system SPOFs? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401084}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, none &lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-02-19
|-
| style=&quot;text-align: center;&quot; | '''5.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does the component protect itself from being overwhelmed by incoming traffic (e.g. rate limiting, load shedding)? If yes, have they been documented? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401085}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Yes&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-22
|-
| style=&quot;text-align: center;&quot; | '''5.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does the component degrade gracefully when its dependencies fail (e.g. timeouts, backoff/retry, fallback/failover, cached or default responses)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401086}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; Yes, see wikitech&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@inga.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-25
|-
| style=&quot;text-align: center;&quot; | '''5.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status|done}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Is the Bus Factor for this service or feature at least 2? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab|T401087}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Three Wikifauna engineers (@bjorn.duck, @inga.duck, @oskar.duck) can deploy and operate Duck end-to-end.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;@bjorn.duck&lt;/span&gt; || style=&quot;text-align: center;&quot; | @jolene.sre || style=&quot;text-align: center;&quot; | 2026-03-28
|}</text>
      <sha1>or61g6973vg5726sjs1xncsw4gt2l3f</sha1>
    </revision>
  </page>
  <page>
    <title>SCROLL/Template</title>
    <ns>0</ns>
    <id>460192</id>
    <revision>
      <id>2435515</id>
      <parentid>2431801</parentid>
      <timestamp>2026-07-12T01:15:55Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>remove DISPLAYTITLE override</comment>
      <origin>2435515</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="31929" sha1="aq5cdu0p42mp8jgp54uaqv6d4fnd8s0" xml:space="preserve">{{SCROLL/Tools/Banner|requested}}
&lt;templatestyles src=&quot;User:Effie_Mouzeli_(WMF)/SCROLL/styles.css&quot; /&gt;

''Service Checklist for Readiness, Operations, Launch and Lifecycle''

__TOC__

{| style=&quot;width: 100%; border: none; background: transparent;&quot;
| style=&quot;vertical-align: top; width: 38%; padding-right: 2%;&quot; |
{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%; border: 1px solid #a2a9b1;&quot;
|-
! colspan=&quot;2&quot; class=&quot;scroll-card-header&quot; | 📜 Service/Component identity
|-
| class=&quot;scroll-card-label&quot; | '''Name'''              || &lt;span class=&quot;scroll-placeholder&quot;&gt;component name&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Type of Request'''      || &lt;span class=&quot;scroll-placeholder&quot;&gt; ⚙️ Service · 🧩 Extension · 🌻 Core Feature · ✨ Feature&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Development Owner'''                || &lt;span class=&quot;scroll-placeholder&quot;&gt;See [https://www.mediawiki.org/wiki/Wikimedia_Production/Service_Catalog/Ownership_Roles_and_Responsibilities#Ownership%20responsibilities Development Owner] for more details. Please add a link to a page with team information.&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''SCROLL bearer'''        || &lt;span class=&quot;scroll-placeholder&quot;&gt;@sre-reviewer&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Soft Launch Target''' (some users)   || &lt;span class=&quot;scroll-placeholder&quot;&gt;YYYY-MM-DD&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Full Launch Target''' (all users) || &lt;span class=&quot;scroll-placeholder&quot;&gt;YYYY-MM-DD&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''SCROLL epic'''          || {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}}
|}

| style=&quot;vertical-align: top; width: 60%;&quot; |
{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%; border: 1px solid #a2a9b1;&quot;
|-
! colspan=&quot;2&quot; class=&quot;scroll-card-header&quot; | Quick references
|-
| class=&quot;scroll-card-label&quot; | '''Phabricator tags'''                   || &lt;span class=&quot;scroll-placeholder&quot;&gt;#tagname&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Wikitech Page URL'''                  || &lt;span class=&quot;scroll-placeholder&quot;&gt;wikitech page&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Google Drive URL'''                   || &lt;span class=&quot;scroll-placeholder&quot;&gt;Drive URL (if applicable)&lt;/span&gt;
|-
| class=&quot;scroll-card-label&quot; | '''Design Document'''                    || &lt;span class=&quot;scroll-placeholder&quot;&gt;link&lt;/span&gt;
|}
|}

{| class=&quot;scroll-legend&quot;
|-
| '''Status:''' &amp;nbsp; 🟢 Done &amp;nbsp;·&amp;nbsp; 🔴 Not Done &amp;nbsp;·&amp;nbsp; '''N/A'''
|-
| '''Priority:''' &amp;nbsp; 🚀 Required for soft launch &amp;nbsp;·&amp;nbsp; 💯 Required for full launch &amp;nbsp;·&amp;nbsp; ❓ Needs scoping / may not be applicable
|-
| '''Required for:''' &amp;nbsp; ⚙️ Service &amp;nbsp;·&amp;nbsp; 🧩 Extension &amp;nbsp;·&amp;nbsp; 🌻 Core Feature &amp;nbsp;·&amp;nbsp; ✨ Feature
|}

== Service Summary ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Documentation
|-
| style=&quot;text-align: center;&quot; | '''1.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have a Wikitech page? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;A Wikitech page is the home for this component's documentation, Most of the items in this list are required to be present in the components Wikitech page.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''1.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Is your service or component present on the [[mw:Wikimedia_Production/Service_Catalog|Service Catalogue]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;The Service Catalogue is the canonical inventory of WMF services. The component should have an entry there.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Contacting The Team
|-
| style=&quot;text-align: center;&quot; | '''1.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are the team's Phabricator tags and contact details documented on the Wikitech page and up to date in officewiki, so the right people can be found promptly when needed? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Wikitech page should include Phabricator tags of the project and team, so bug reports are routed to the right people. Moreover, it should be clear in the wikitech page how to contact the team, or ensure there is a link pointing to the team's page. Lastly, it would be great if we are certain the team's office page is up to date.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | External Reviews
|-
| style=&quot;text-align: center;&quot; | '''1.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by SRE? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Reach out to SRE early, both to agree target dates (soft launch, full launch) and to walk through the checklist together so you can confirm which items are relevant and which can be skipped.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE &amp;amp; Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''1.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by Security? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;The Security team is aware of this work and has communicated their requirements (if applicable).&lt;/span&gt; || style=&quot;text-align: center;&quot; |  || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''1.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Has the design been reviewed by Data Persistence? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Data Persistence is aware of this work and has communicated their requirements (if applicable).&lt;/span&gt; || style=&quot;text-align: center;&quot; |  || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|}

== Operating Procedures ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Interactions
|-
| style=&quot;text-align: center;&quot; | '''2.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have you described how the service/feature interacts with common mediawiki userflows on wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Engineers should be able to read the Wikitech page and understand where this component sits on the critical path and assess the impact when something goes wrong.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Infrastructure
|-
| style=&quot;text-align: center;&quot; | '''2.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service runs on baremetal, are the server prefixes documented on wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;If the service runs on baremetal or VMs, document the server prefixes on Wikitech.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''2.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service runs on Kubernetes, has it been added to [[Kubernetes/Deployment Charts|deployment charts]] (helm charts) and has the relevant [[Kubernetes/Service ports|kubernetes service defined]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;A Kubernetes deployment needs a Helm chart in deployment-charts, and a Kubernetes Service if it accepts traffic from outside the cluster.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE &amp;amp; Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''2.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  If you have a Service URL? should it be present in service.yaml on Puppet? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;A standalone service accepting traffic must exist in Puppet's service.yaml; document the URL where the service can be reached.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''2.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀❓ || style=&quot;text-align: center;&quot; | ⚙️ ||  Does your service have a staging environment? if yes, is it documented on Wikitech? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;If the service has a staging environment, document its URL on Wikitech.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Service Traffic
|-
| style=&quot;text-align: center;&quot; | '''2.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Do you have an estimation of the traffic your service will be serving? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Teams should be able to work out an estimation of the traffic they expect, and the methodology used. If that is not straightforward, reach out to SRE and we can work through it together.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''2.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Does your service accept traffic directly from the CDN? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;If the service has public endpoints, SRE Traffic may need to provide additional configuration for routing and caching.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE &amp;amp; Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''2.8''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  If your service is in discovery, can the service be depooled safely and run from a single DC, any special consideration for [[Switch Datacenter|DC Switchover]]? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;An active/active or active/passive service needs a discovery URL; confirm it can be depooled to a single datacentre without user-visible impact.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE &amp;amp; Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''2.9''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Have you documented with which systems/datastores your service needs to communicate with? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;A clear list of dependencies helps with capacity planning as well as monitoring.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Maintenance
|-
| style=&quot;text-align: center;&quot; | '''2.10''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  If you have dependencies on maintenance scripts (mw-script) or crons (mw-cron), have they been documented and recently tested? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Maintenance scripts and crons often go untested for long periods. Documenting and testing them prevents surprises when they fail or need to be re-run.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|}

== Release Confidence ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Building and Testing
|-
| style=&quot;text-align: center;&quot; | '''3.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does your wikitech page link to the code repository and production branch, and recent image (if applicable)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Link the code repository and the production branch from your Wikitech page, and document the name and location of the most recent container image.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''3.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have integration and unit tests (CI)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Unit and integration tests in CI give confidence that changes are safe to deploy.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Initial Rollout
|-
| style=&quot;text-align: center;&quot; | '''3.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have a rollout plan (soft and full launch), including the common failure points you have identified for launch day? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;A documented rollout plan for soft and full launch, covering the deployment sequence, smoke tests, rollback, communication, and the common failure points identified for launch day (eg cold caches or an overwhelmed dependency).&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE &amp;amp; Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Deploying to Production
|-
| style=&quot;text-align: center;&quot; | '''3.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you have Configuration &amp; Deployment Owners? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;See [https://www.mediawiki.org/wiki/Wikimedia_Production/Service_Catalog/Ownership_Roles_and_Responsibilities#Ownership%20responsibilities  Configuration &amp; Deployment Owners] for more details. Please ensure this is also documented on Wikitechready and authorised to deploy means you can roll out changes and fixes on your own schedule, rather than waiting for help.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|}

== Observability ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Metrics, Instrumentation, Logging
|-
| style=&quot;text-align: center;&quot; | '''4.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are you exporting Prometheus metrics and sending logs to Logstash? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Prometheus and Logstash are WMF's standard tools for metrics and logs. Exporting to both is the baseline for any observable service.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''4.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are key user flows and business metrics instrumented and exported? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Instrumenting user-facing flows and business outcomes helps measure what matters to users and assess impact during incidents.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Service Level Objectives
|-
| style=&quot;text-align: center;&quot; | '''4.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have SLOs been agreed on and published to assist in evaluating the impact on end users? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;SLOs should be agreed on and published so the team can evaluate user-facing impact and reason about the error budget.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''4.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Have the SLO dashboard and alerts been configured in Sloth? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Configure the SLO dashboard and alerts in Sloth.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Monitoring
|-
| style=&quot;text-align: center;&quot; | '''4.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are your dashboards (Grafana, Logstash) documented on  Wikitech page, and include links to related dashboards and runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Document your Grafana and Logstash dashboards on the Wikitech page. The dashboards themselves should have cross-links to related dashboards, dependencies and other information.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''4.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯❓ || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Do you monitor latency variations at the p50, p75, and p99 percentiles (eg via envoy, or other business metrics), as well as the status of external dependencies? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Monitor latency at p50/p75/p99 (eg via Envoy or business metrics) and the status of any external dependencies.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Alerting
|-
| style=&quot;text-align: center;&quot; | '''4.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Are your alerts documented on Wikitech and committed to [[Alertmanager]]: paging alerts, alerts for excessive errors (business, infrastructure, budget burn), and linked to runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Document your alerts on Wikitech and commit them to Alertmanager: paging alerts, alerts for excessive errors (business, infrastructure, budget burn), each linked to a runbook.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;SRE &amp;amp; Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Incident Response
|-
| style=&quot;text-align: center;&quot; | '''4.8''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does your component have and Incident Response owner and Runbooks? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt; See [https://www.mediawiki.org/wiki/Wikimedia_Production/Service_Catalog/Ownership_Roles_and_Responsibilities#Ownership%20responsibilities Incident Response Owner] for more details and ensure this is documented on wikitech. A runbook documents common operational procedures and the responses to each alert. Check [[:Category:Runbooks]]&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|}

== Reliability and performance ==

{| class=&quot;wikitable&quot; style=&quot;font-size: 90%; width: 100%;&quot;
|-
! style=&quot;width: 4%;&quot; class=&quot;scroll-col-header-center&quot; | #
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Status
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Priority
!! style=&quot;width: 10%;&quot; class=&quot;scroll-col-header-center&quot; | Required for
!! class=&quot;scroll-col-header&quot; | Item
!! style=&quot;width: 7%;&quot; class=&quot;scroll-col-header-center&quot; | Phab
!! class=&quot;scroll-col-header&quot; | Remarks
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Assignee
!! style=&quot;width: 8%;&quot; class=&quot;scroll-col-header-center&quot; | Signed off by
!! style=&quot;width: 6%;&quot; class=&quot;scroll-col-header-center&quot; | Date
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Resources
|-
| style=&quot;text-align: center;&quot; | '''5.1''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️ ||  Do you have an estimation of the resources you will need? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Estimated CPU, memory, and storage requirements drive capacity planning. Linked phab task with discussion.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''5.2''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️ ||  Is the service designed to scale up or down as needed? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;SRE should be able to add or remove resources on demand without contacting the team.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| colspan=&quot;10&quot; class=&quot;scroll-section&quot; | Reliability
|-
| style=&quot;text-align: center;&quot; | '''5.3''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Does each component have its own health or liveness check to ensure production traffic does not reach an unhealthy endpoint? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Liveness and readiness checks should be in place for routing and alerting purposes.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''5.4''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 🚀 || style=&quot;text-align: center;&quot; | ⚙️ ||  Have you identified your system SPOFs? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Single Points of Failure are components whose loss takes down the service. Identifying them is the first step to mitigating or accepting the risk. Should be documented on wikitech&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''5.5''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does the component protect itself from being overwhelmed by incoming traffic (e.g. rate limiting, load shedding)? If yes, have they been documented? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Protect the service from being overwhelmed by incoming traffic (eg rate limiting, load shedding), and document these protections.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''5.6''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Does the component degrade gracefully when its dependencies fail (e.g. timeouts, backoff/retry, fallback/failover, cached or default responses)? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;Well-defined timeout, retry, and fallback behaviour keeps the component working when its dependencies misbehave.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|-
| style=&quot;text-align: center;&quot; | '''5.7''' || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/status}} || style=&quot;text-align: center;&quot; | 💯 || style=&quot;text-align: center;&quot; | ⚙️🧩🌻✨ ||  Is the Bus Factor for this service or feature at least 2? || style=&quot;text-align: center;&quot; | {{User:Effie_Mouzeli_(WMF)/SCROLL/phab}} ||  &lt;span class=&quot;scroll-remarks&quot;&gt;At least two people should understand the service well enough to ensure its operation and longevity.&lt;/span&gt; || style=&quot;text-align: center;&quot; | &lt;span class=&quot;scroll-assignee&quot;&gt;Dev Team&lt;/span&gt; || style=&quot;text-align: center;&quot; | || style=&quot;text-align: center;&quot; |
|}</text>
      <sha1>aq5cdu0p42mp8jgp54uaqv6d4fnd8s0</sha1>
    </revision>
  </page>
  <page>
    <title>Blazegraph Migration: Rewrite of Label and Utility Services and Functions</title>
    <ns>0</ns>
    <id>460376</id>
    <revision>
      <id>2435512</id>
      <parentid>2434699</parentid>
      <timestamp>2026-07-12T01:06:59Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;sparql&quot;</comment>
      <origin>2435512</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="35777" sha1="h564gjpbx0fyh2ncw7nn2o7ghyvjsij" xml:space="preserve">This Wikitech page catalogs the rewrites needed to translate several utility query services provided by Blazegraph into portable, standards-conformant SPARQL 1.1. To port these queries, each Blazegraph-specific construct must be substituted with an equivalent SPARQL 1.1 pattern, restructured into a portable form, or (where no equivalent exists) removed with an explanatory note. 

The rewrites described in this document cover six such constructs: 
* The wikibase:label service 
* Named subqueries 
* Inline query hints 
* bd:sample and bd:slice result-limiting predicates
* The wikibase:decodeURI function 
* The wikibase:label service  

For each construct, the sections that follow explain its Blazegraph-specific semantics and specify the rewrite.

== wikibase:label ==
The wikibase:label service auto-resolves an entity's label, description, and aliases into result variables, honoring a language priority list with fallback. There are two modes used to define variable names - a manual and an automatic mode.

In the manual mode, the query writer declares the variable names. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   SERVICE wikibase:label {
       bd:serviceParam wikibase:language &quot;de,en&quot; .
       ?item rdfs:label ?label .
       ?item schema:description ?description .
       ?item skos:altLabel ?alternate . }
}
&lt;/syntaxhighlight&gt;

This query produces results as shown below:
[[File:Rewrite Label-Fig 1.png|Figure 1]]

In automatic mode, no names are defined and the service projects specific names for any variables, ?x. The projected names are ?xLabel, ?xDescription, ?xAltLabel. A query could be written as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel WHERE {
   ?item wdt:P31 wd:Q515 .
   SERVICE wikibase:label { 
       bd:serviceParam wikibase:language &quot;[AUTO_LANGUAGE],de,en&quot;. }
}
&lt;/syntaxhighlight&gt;

This query produces the same results but with different names for the label, description and alias outputs.

When rewriting a query to produce equivalent results, the rewrite must consider:
* Language priority + fallback — wikibase:language is a comma-separated priority list where the first available language wins
** Usage of [AUTO_LANGUAGE] will be continued as a UI-only feature (it expands the list of languages to include the requester's UI/browser language)
* mul fallback — a label or alias under the multilingual code “mul” applies to all languages and is consulted after the listed languages
** There are no “mul” descriptions 
* Allowance for unbound labels, descriptions and aliases
* QID fallback for labels (only)
** If no label exists in any requested language or mul, the entity ID (e.g., Q42) is returned 
** There is no fallback for descriptions or altLabels (they are unbound if not found)
* skos:altLabel can be multivalued
** label service returns all aliases of the best language as one comma-joined string
* Applicable to both items and properties 

At its most basic, the rewrite must drop the service call, determine each entity variable and its related label/description/alias variable names, and then return those variable values based on SPARQL OPTIONAL clauses. 

Why SPARQL OPTIONALs? Because defining a raw triple pattern (such as ?item schema:description ?description) without OPTIONAL requires the item to have a description. If one is not defined, then the entity will be deleted from the result set. This is not the behavior of the wikibase:label service.

To rewrite the first query in this section, a naive approach (but one that illustrates what is happening “under the covers”) produces the following query:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   # --- label: priority de &gt; en &gt; mul, else QID ---
   OPTIONAL { ?item rdfs:label ?label_de  . FILTER(LANG(?label_de)  = &quot;de&quot;)  }
   OPTIONAL { ?item rdfs:label ?label_en  . FILTER(LANG(?label_en)  = &quot;en&quot;)  }
   OPTIONAL { ?item rdfs:label ?label_mul . FILTER(LANG(?label_mul) = &quot;mul&quot;) }
   BIND(COALESCE(STR(?label_de), STR(?label_en), STR(?label_mul),
                 STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?label)
   # --- description: priority de &gt; en &gt; mul, else unbound ---
   OPTIONAL { ?item schema:description ?desc_de  . FILTER(LANG(?desc_de)  = &quot;de&quot;)  }
   OPTIONAL { ?item schema:description ?desc_en  . FILTER(LANG(?desc_en)  = &quot;en&quot;)  }     
   BIND(COALESCE(STR(?desc_de), STR(?desc_en)) AS ?description)
   # --- aliases: priority de &gt; en, multi-valued → GROUP_CONCAT per language ---
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_de) WHERE {
         ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;de&quot;)
      } GROUP BY ?item
   }
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_en) WHERE {
        ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;en&quot;)
      } GROUP BY ?item
   }
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_mul) WHERE {
        ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;mul&quot;)
      } GROUP BY ?item
   }
   BIND(COALESCE(?alt_de, ?alt_en, ?alt_mul) AS ?alternate)
}
&lt;/syntaxhighlight&gt;

There are several important things to note:
* All the resulting label/description labels are COALESCE’d as STR(?x) in order to remove the language tags
** This is needed since wikibase:label does not return language tags, but plain literals
** This is not needed for the aliases since GROUP_CONCAT returns a plain literal
* “mul” must be explicitly added as a language
* The COALESCE order is defined by the language priority with “mul” as the lowest priority
* The label’s COALESCE falls back to the entity QID if there are no labels in the specified languages
* Since aliases are multi-valued, each language’s results are GROUP_CONCAT'ed in their own GROUP BY ?item subqueries, then COALESCE'd for language priority
** There is a performance implication for this - the subqueries (SELECT clauses within the OPTIONALs) are always executed first and will concatenate alias names across all items with “de” or “en” aliases 
*** This is because there are no binding constraints on ?item
** To improve the performance, the constraining triple(s) for ?item (in this case, &lt;i&gt;?item wdt:P31 wd:Q515&lt;/i&gt;) should be moved inside the sub-SELECT scope

Executing the rewritten query obtains the following (equivalent but with different order) results:
[[File:Rewrite Label-Fig 2.png|Figure 2]]

The rewritten query was characterized as “naive” since it included the COALESCE. That forces an explicit ordering of the variable binding. However, it is not needed since the OPTIONAL clause is a left join. Once the first OPTIONAL binds ?label to the German label, the same variable in the next OPTIONAL's triple pattern is no longer a fresh binding. It becomes a join constraint: ?item rdfs:label ?label must now match the already-bound German literal, and FILTER(LANG(?label) = &quot;en&quot;) on a @de literal is false. Therefore, the “en” OPTIONAL adds nothing and the German value survives. If a German label did not exist (?label is unbound), the English OPTIONAL is free to bind it; then mul; and so on. The left joins themselves implement &quot;first match wins&quot; which is what COALESCE was doing (making it redundant).

What remains is only the QID fallback for ?label and stripping the language tag. That can be written without COALESCE:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
BIND(IF(BOUND(?label), STR(?label), STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?itemLabel)
&lt;/syntaxhighlight&gt;

Given this discussion, the rewritten query becomes:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   # label: de &gt; en &gt; mul into one shared ?lbl (left joins give priority), else QID
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;de&quot;)  }
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;en&quot;)  }
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;mul&quot;) }
   BIND(IF(BOUND(?lbl), STR(?lbl), STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?label)
   # description: de &gt; en into shared ?dsc, no mul and no fallback; 
   # STR(?dsc) to remove lang tag
   OPTIONAL { ?item schema:description ?dsc . FILTER(LANG(?dsc) = &quot;de&quot;) }
   OPTIONAL { ?item schema:description ?dsc . FILTER(LANG(?dsc) = &quot;en&quot;) }
   BIND(STR(?dsc) AS ?description)
   # aliases: de &gt; en &gt; mul into shared ?alternate, constrained, GROUP_CONCAT
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;de&quot;)  
       } GROUP BY ?item }
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;en&quot;)  
       } GROUP BY ?item }
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;mul&quot;) 
       } GROUP BY ?item }
}
&lt;/syntaxhighlight&gt;

The rewrite for the second query in this section (using automatic variable name binding and AUTO-LANGUAGE) uses the same overall pattern. But, there are a few items to note:
* The AUTO_LANGUAGE UI token will be maintained - so, it can be substituted as a language directly in the rewritten query (e.g., LANG(?x) = “AUTO_LANGUAGE”)
** Outside of the UI, there is no equivalent concept - all languages must be explicitly declared in the query
* If the query projected additional entity variables beyond ?item (e.g. ?country), each would need its own blocks for determining ?countryLabel, ?countryDescription and ?countryAlias 

=== Label Service Results for an Unbound Variable ===

It is possible that the variable whose label is reported is not bound when the label results are returned. An example is a request for all plumbers' names and their mothers' names. Not all the plumbers' names are defined and many of their mothers are unknown. 

For Blazegraph, this query would be written:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt; 
SELECT ?a ?aLabel ?b ?bLabel WHERE {
  ?a wdt:P31 wd:Q5 .
  ?a wdt:P106 wd:Q252924 .
  OPTIONAL { ?a wdt:P25 ?b } 
  SERVICE wikibase:label { 
       bd:serviceParam wikibase:language &quot;en&quot;. }
}
&lt;/syntaxhighlight&gt;

Using the rewrite pattern currently defined, the rewrite would appear as:
&lt;syntaxhighlight&gt; 
SELECT ?a ?aLabel ?b ?bLabel WHERE {
  ?a wdt:P31 wd:Q5 .
  ?a wdt:P106 wd:Q252924 .
  OPTIONAL { ?a wdt:P25 ?b } 
  OPTIONAL { ?a rdfs:label ?aLabel . FILTER ( lang(?aLabel) = 'en' ) }
  OPTIONAL { ?b rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) }
}
&lt;/syntaxhighlight&gt;

But, this query likely times out. Why? Each OPTIONAL clause is executed after &quot;pushing down&quot; all currently bound variables. The variable, ?a, is clearly bound by the triples in the second and third lines. So, the evaluation of the OPTIONAL clause for ?a's label is efficient. However, the variable, ?b, becomes bound in the &lt;i&gt;OPTIONAL { ?a wdt:P25 ?b }&lt;/i&gt; clause, but that is not visible to the clause requesting ?b's label. The clause, &lt;i&gt;OPTIONAL { ?b rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) }&lt;/i&gt;, becomes a request for all possible entities and their labels. Then, this is resolved using a left join across the OPTIONALs' results.

An approach to correct this is to check that the variable is bound before executing the OPTIONAL clause to retrieve its label. Taking this approach, the query would be rewritten as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt; 
SELECT ?a ?aLabel ?b ?bLabel WHERE {
  ?a wdt:P31 wd:Q5 .
  ?a wdt:P106 wd:Q252924 .
  OPTIONAL { ?a wdt:P25 ?b } 
  BIND(IF(BOUND(?a), ?a, &quot;unbound&quot;) AS ?al)
  OPTIONAL { ?al rdfs:label ?aLabel . FILTER ( lang(?aLabel) = 'en' ) }
  BIND(IF(BOUND(?b), ?b, &quot;unbound&quot;) AS ?bl)
  OPTIONAL { ?bl rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) }
}
&lt;/syntaxhighlight&gt;

This works as desired, returning a set of results equivalent to that from Blazegraph.

Clearly, there are multiple ways to modify the query above. 
# Since ?a is clearly bound, the IF(BOUND(?a), ...) check is not needed
# Or, the first OPTIONAL clause could be rewritten as:
::&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
OPTIONAL { ?a wdt:P25 ?b .
   OPTIONAL { ?b rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) } }
&lt;/syntaxhighlight&gt;

However, a straightforward and simple BIND(IF(BOUND(?x), ... check requires less analysis and is still performant.

=== Query Rewrite Pattern for wikibase:label ===
Collecting all the requirements and considerations above, the rewrite pattern can be explained as:
* Determine the languages to be output, splitting wikibase:language values using a comma delimiter
** And their priority order
* Determine the names of the projected variables 
** Which then defines the types of OPTIONAL “blocks” needed - label/description/alias
* Create those blocks based on the pattern shown in the last query above (BIND checking for a bound variable and then the OPTIONAL clauses requesting the label in the specified and prioritized languages), and substitute them in the query in place of the wikibase:label service
** For the &lt;i&gt;alias&lt;/i&gt; labels, taking care to re-insert the constraining triple(s) for the variable(s) in the OPTIONAL clauses' SELECT subqueries' body 
*** This is needed since those SELECT subqueries are executed first and their results used in the OPTIONAL clauses left-join operation at the end of processing

== Named Subqueries ==
Although SPARQL supports nested subqueries, it does not support assigning them a name or specifically “including” them at specific (sometimes multiple) points in a scoping query. Blazegraph allows this using the keywords, INCLUDE, AS and WITH. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer
WITH {
  SELECT ?person WHERE {
    ?person wdt:P31 wd:Q5 ; wdt:P106 wd:Q901 .   # humans who are scientists
  }
} AS %scientists
WHERE {
  INCLUDE %scientists .
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

The query first executes the &lt;i&gt;WITH { … } AS %name&lt;/i&gt; subquery, and does this only once. Its results are materialized into a named temporary set, and every &lt;i&gt;INCLUDE %name&lt;/i&gt; joins against that same cached set. It's effectively a SQL Common Table Expression (CTE) using WITH … AS. 

Named subqueries serve three purposes: 
* Clarity - cleanly defining the query
* Reuse - allowing an expensive subquery to be executed once and used in several places 
* Join-order control - forcing the subquery to run first, rather than letting the query optimizer to insert it

Unfortunately, SPARQL 1.1 has no equivalent to named subqueries. The portable equivalent is an inline sub-SELECT that duplicates the code of the WITH clause and replaces each &lt;i&gt;INCLUDE %name&lt;/i&gt; with the body of the sub-SELECT. For the example above:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer WHERE {
  { SELECT ?person WHERE {
      ?person wdt:P31 wd:Q5 ; wdt:P106 wd:Q901 .
  } }
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

The most important question is whether the subqueries are simply copied wherever they are referenced. And, for ordinary, deterministic queries, the answer is yes. The sub-SELECT body can be pasted at each INCLUDE site&lt;ref&gt;This is actually a worst case rewrite since the pattern can be simplified to only paste within each unique query scope. This is discussed in more detail in the Query Rewrite sub-section.&lt;/ref&gt;.

Again, unfortunately, &quot;simply copied&quot; hides three problems:  
* Complexity, when a sub-query is copied multiple times and the query text “explodes” 
* Performance, since the compute-once guarantee is lost
** Blazegraph evaluates the block once regardless of the number of references to it
** Instead, if it is inlined at N sites, then the query is evaluated N times
** For a “cheap” subquery, this is mainly irrelevant
** For an “expensive” subquery referenced several times, a large performance cost could be incurred
* Correctness, when the sub-query is non-deterministic or mints blank nodes AND occurs multiple times
** Non-deterministic queries include functions such as RAND() and UUID()/STRUUID()
*** When Blazegraph executes the sub-query once, every INCLUDE sees the same bound variables and values
*** Instead, if inlined multiple times, each copy re-evaluates those functions and returns different values
** The sub-query may return variables as blank nodes
*** As above, when Blazegraph executes the sub-query once, every INCLUDE sees the same bound variables and bnode values
*** Instead, if inlined multiple times, each copy returns different bnode identities
** Other examples of non-deterministic results are subqueries that use:
*** A LIMIT but no ORDER BY, making the result selection defined at execution time
*** GROUP_CONCAT, since the SPARQL aggregate functions operate over unordered sets
*** SAMPLE()
*** A federated service call to an external endpoint
**** If there are N separate calls to the federated service, results could vary between the calls due to changes in the data, a LIMIT on the federated result, or other non-deterministic behavior at the endpoint
*** Blazegraph-specific sampling utilities such as bd:slice and bd:sample

If non-deterministic or blank node results are included in the solution set, &lt;i&gt;AND the named subquery is used more than once&lt;/i&gt;, it cannot just be copied but must first be executed/materialized. Then, its results can be manually added back into the query as a VALUES block.

For example, rewriting the scientist query above using the VALUES approach (even though this is deterministic), results in the following: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer WHERE {
  VALUES ?person { wd:Q4118502 wd:Q100156581 … }   # Results of running %scientists once
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

There are a few obvious problems with the VALUES approach. First, it is a two-step manual or programmatic process. Second, it is cumbersome when the result set is large (thousands of results).

=== Query Rewrite Pattern for Named Subqueries ===
For rewriting a named subquery, its contents (within the WITH { … } AS clause) are replaced by an inlined &lt;i&gt;{ SELECT variables WHERE { … } }&lt;/i&gt; clause. If there is only a single reference to the sub-query, the result of inlining is equivalent to the original query. 

However, questions arise when there are multiple references to the sub-query and these occur in multiple scopes&lt;ref&gt;Examples of different scopes in a SPARQL query are having nested sub-queries or separate UNION branches.&lt;/ref&gt;. One solution to this is to rewrite the sub-query in each scope. Further complications then arise only when the results of the sub-query are non-deterministic and this is visible in the externalized / observable result variables. 

There is complexity in determining these two characteristics (non-deterministic and projected). To determine them, the query’s algebra tree&lt;ref&gt;Obtained using Jena ARQ or rdflib libraries&lt;/ref&gt; must be analyzed. This can be used to first define whether multiple scopes are involved. If the enclosing graph pattern for each reference to a named subquery is the same - then there is only one scope (and no rewrite problems). However, if two INCLUDEs have different enclosing patterns, then their potential results need to be examined. 

A simple way to determine if the results are non-deterministic due to generated blank nodes is to actually run the sub-query and see if any results are reported with type = “bnode”&lt;ref&gt;It is important to distinguish Wikidata skolemized blank nodes, and query engine-minted nodes. Wikidata’s blank nodes will be returned in a query binding as type=&quot;uri&quot; with a specific, consistent IRI value.&lt;/ref&gt;. If so and the subquery is referenced in multiple scopes, then that query is not eligible for rewriting.

Other simple checks for non-deterministic results is whether the subquery includes bd:slice or bd:sample (which have approximate rewrites), or a LIMIT without an ORDER BY. 

Lastly, if the subquery uses SAMPLE(), RAND(), UUID(), STRUUID() or GROUP_CONCAT(), and if that use occurs in multiple scopes, and if the results of the function calls are observable in the results, then the query is &lt;b&gt;ineligible&lt;/b&gt; for rewrite. Being “observable” occurs if the functions appear in the SELECT clause, affect which result sets are returned (used in FILTER/HAVING or ORDER BY under a LIMIT/OFFSET), or feed a projected BIND. 

Examples of these are:
* A SELECT variable (for example, SELECT ?k (SAMPLE(?v) AS ?s))
* A FILTER/HAVING clause (e.g., HAVING(SAMPLE(?v) &gt; 5))
* An ORDER BY (SAMPLE(?v)) ... LIMIT n clause (since the results which “clear” the LIMIT are dependent on the SAMPLE selection) 

=== What about QLever’s Materialized Views? ===
QLever has defined a feature (still in beta at the time of writing) called [https://docs.qlever.dev/materialized-views/ materialized views] which seems similar to Blazegraph’s named subqueries. Both provide the capability to create a named, compute-once result set that is referenced by name. However, there are several differences and explicit issues. The most important of these are: 
* SPARQL 1.1 compliance:
** Neither Blazegraph named subqueries nor QLever materialized views are SPARQL 1.1 compliant
** Also, the creation of a materialized view requires server-level access (not allowed on WDQS)
* Definition location: 
** A named subquery (WITH … AS %x) is defined inside the query itself - it is self-contained and recomputed every run
** A materialized view is created out of band and stored on disk (dependent on server-side state) 
* Lifetime and freshness:
** Since named subqueries are recomputed on every query execution, they are never stale
** Materialized views are persistent and must be recreated if they go stale 

== Query Hints ==
QLever does not expose a Blazegraph-style query-hint system and this is not a part of the SPARQL 1.1 standard. So, the rewrite pattern for Blazegraph hint:* directives is to strip them entirely.  

It is important to note that QLever's optimizer can be influenced by the structure of a SPARQL query. It is recommended to use subqueries to force early evaluation of a portion of a query. When enclosing a block inside a SPARQL subquery (SELECT ... WHERE { ... }), QLever will execute that block before joining its results with the remainder of the query.

== bd:sample ==
bd:sample is a Blazegraph service that selects a bounded subset of solutions from a graph pattern &lt;i&gt;without&lt;/i&gt; fully materializing and sorting the entire set of results. Its purpose is to keep otherwise-expensive queries under the WDQS timeout by working on a sample instead of the full data. 

For example, the following query returns 2000 “locations” (&lt;i&gt;wdt:P276 ?value&lt;/i&gt;) and orders them by the number of distinct ?items that reference them. (Note however, that this is the ordering within the sample set of results and not across Wikidata.)

&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?value (COUNT(DISTINCT ?item) as ?count) WHERE {
 SERVICE bd:sample {
   ?item wdt:P276 ?value .
   bd:serviceParam bd:sample.limit 2000 . }
} GROUP BY ?value
  ORDER BY DESC(COUNT(DISTINCT ?item))
&lt;/syntaxhighlight&gt;

The query pattern defined within the service binds the relevant variables (in this case, ?item and ?value). These can then be used in the surrounding SPARQL.

The possible service parameters are:
{| class=&quot;wikitable&quot;
|-
! Parameter !! Usage !! Default
|-
| bd:sample.limit || Maximum number of result sets to return || 100
|-
| bd:sample.sampleType || “RANDOM”, “EVEN” (evenly spaced sample), “DENSE” (first N) || RANDOM
|-
| bd:sample.seed || RANDOM's seed || 0 
|}

Rewriting the service depends on the sampleType parameter, but none of the rewrites preserve the performance characteristics of bd:sample. For all rewrites, the engine must evaluate the inner pattern first and run to completion. 

The closest SPARQL 1.1 query to that shown above is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?value (COUNT(DISTINCT ?item) AS ?count) WHERE {
  { # Translate the query pattern within the service to a SELECT query
    SELECT ?item ?value WHERE {    
      ?item wdt:P276 ?value .
    } ORDER BY RAND()      # Default is RANDOM, so apply that ordering
      LIMIT 2000           # LIMIT 2000
  }
} GROUP BY ?value
  ORDER BY DESC(COUNT(DISTINCT ?item))
&lt;/syntaxhighlight&gt;

The above query does complete on QLever, but a query that returns a larger result set (for example, asking for a subset of humans, Q5) or is more complex could exceed the 60 second timeout. In all cases, applying a LIMIT to the inner query is needed to force a cut-off. That is also why bd:sample has a limit parameter.

It is valuable to note that the SPARQL 1.1 query could be changed to remove the ORDER BY RAND() statement and just use LIMIT. That would definitely improve the performance, and would mimic choosing the “DENSE” sampleType of bd:sample. 

“EVEN” is the last sampleType value that could be used. To replicate the behavior requires positional/windowing logic that standard SPARQL 1.1 cannot express. However, the result can be approximated with the RANDOM rewrite above (a uniform random sample is usually an acceptable stand-in for an evenly-spaced one).

There is another service parameter that we did not discuss - seed. This would be used for reproducibility. Here is a Blazegraph query using that parameter:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  SERVICE bd:sample {
    ?item wdt:P31 wd:Q5 .                         # ?item is an instance of human
    bd:serviceParam bd:sample.limit 1000 .
    bd:serviceParam bd:sample.sampleType &quot;RANDOM&quot; .
    bd:serviceParam bd:sample.seed 42 .           # fixed seed → same 1000 every run
  }
}
&lt;/syntaxhighlight&gt;

The rewrite is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  { SELECT ?item WHERE {
      ?item wdt:P31 wd:Q5 .
    } ORDER BY MD5(CONCAT(&quot;42&quot;, STR(?item)))   # &quot;42&quot; plays the role of bd:sample.seed
      LIMIT 1000
               }
}
&lt;/syntaxhighlight&gt;

Note that the above query can successfully execute on a QLever server in under 60 seconds. So, simply having a large intermediate result set does not necessarily equate to query timeout.

=== Query Rewrite Pattern for bd:sample ===
These steps summarize the rewrite patterns for bd:sample:
* Lift the query &lt;pattern&gt; out of SERVICE bd:sample {…}  and rewrite it as an inner SELECT … WHERE { &lt;pattern&gt; }, projecting the variables present in the outer query 
* Append LIMIT N (the bd:sample.limit or use the default, 100, if it was omitted)
* Add an ORDER BY if the sampleType is “RANDOM” or “EVEN” 
** RAND() if a seed is not specified
** MD5(CONCAT(seed, STR(?key))) otherwise
* Leave the surrounding query (GROUP BY, COUNT, etc.) unchanged

== bd:slice ==
bd:slice is a Blazegraph service that provides efficient, stable pagination over a single triple pattern. It walks through a very large result set in deterministic chunks without re-sorting on each request. Its purpose is to keep otherwise-expensive queries under the WDQS timeout by working on a subset instead of the full data. 

The most “mechanically” equivalent SPARQL 1.1 pattern is to perform the query with an ORDER BY, and OFFSET / LIMIT. However, this becomes increasingly non-performant as the OFFSET size grows - since OFFSET skips rows in the result set by first generating and then discarding them. 

Another way to rewrite the query using SPARQL 1.1 is to create an “index” and use that to segment the results. Taking this approach each page carries a similar performance cost. 

For example, the following Blazegraph query is used to return result sets of 10000 humans. Subsequent calls increase the offset by 10000 and add 10000 to the limit. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
 SERVICE bd:slice {
   ?item wdt:P31 wd:Q5 .                    # single triple pattern (required)
   bd:serviceParam bd:slice.offset 0 .
   bd:serviceParam bd:slice.limit 10000 .
 }
}
&lt;/syntaxhighlight&gt;

To rewrite this query using SPARQL 1.1 and a query-relevant index requires some manual construction … The largest value of the value from a run is inserted into a FILTER on the index.

For the query above, the first iteration is written as: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .             
} ORDER BY (str(?item))
  LIMIT 10000  
&lt;/syntaxhighlight&gt;

The second iteration becomes: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .
  FILTER(STR(?item) &gt; &quot;http://www.wikidata.org/entity/Q100310806&quot;)
} ORDER BY STR(?item)
  LIMIT 10000 
&lt;/syntaxhighlight&gt;

Where the entity, Q100310806, is the “last” ?item output by the previous iteration.

It is important to note that because of the indexes and performance characteristics of QLever, it is also possible to execute the query unconstrained  by order or limits. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .
}  
&lt;/syntaxhighlight&gt;

A total of 13M+ results are output, and indicates that bd:slice was a performance optimization required by Blazegraph as opposed to addressing a query-related need.

As for bd:sample, there is another service parameter for bd:slice that should be discussed - range. In range mode, bd:slice doesn't return rows of results, but returns a single value - a count of the solutions that is computed from index boundaries. A variable is named and bd:slice.range estimates a count. Note that word, estimates. This is problematic since it is usually an overestimate and done as another performance optimization.

Consider these two Blazegraph queries:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?range WHERE {
  SERVICE bd:slice {
    ?item wdt:P569 ?dob .                       # single triple pattern
    bd:serviceParam bd:slice.range ?range .     # ?range ← fast index count of matches
  }
}
&lt;/syntaxhighlight&gt;

And:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?range WHERE {
  SERVICE bd:slice {
    ?item wdt:P569 ?dob .
    FILTER(?dob &gt;= &quot;1900-01-01&quot;^^xsd:dateTime &amp;&amp; ?dob &lt; &quot;2000-01-01&quot;^^xsd:dateTime)
    bd:serviceParam bd:slice.range ?range .
  }
}
&lt;/syntaxhighlight&gt;

Both of these queries return a ?range of approximately 8074600 (varying +/- 3).

However, a corresponding query using SPARQL 1.1, gives exact results.
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT (COUNT(*) AS ?range) WHERE {
  ?item wdt:P569 ?dob .
  FILTER(?dob &gt;= &quot;1900-01-01T00:00:00&quot;^^xsd:dateTime &amp;&amp; 
               ?dob &lt; &quot;2000-01-01T00:00:00&quot;^^xsd:dateTime)
}
&lt;/syntaxhighlight&gt;

Which returns a count of 5457644. With the FILTER removed, the count is 8074572 (pretty close to the overall Blazegraph estimate!).

=== Query Rewrite Pattern for bd:slice ===
The following list defines the rewrite patterns for bd:slice with limit and offset, or with range:
* For limit/offset:
** Evaluate first if limit/offset are actually needed. For many queries, QLever’s indexes and performance are sufficient to remove the need for this optimization.
** If the query does timeout, then pagination is needed. The simplest portable form is to use ORDER BY &lt;key&gt; + OFFSET + LIMIT, bumping OFFSET with each query request. But, note that OFFSET cost grows with depth. 
*** For deep paging, it is recommended to use keyset pagination. Pick a deterministic sort key (e.g. STR(?subject)), ORDER BY it, issue an initial LIMIT query, then for each subsequent page add &lt;i&gt;FILTER(STR(?subject) &gt; “last_value_from_previous_query”)&lt;/i&gt; with the same ORDER BY and LIMIT. 
*** Keyset replaces the growing OFFSET with an index seek
*** Remember - do not use both OFFSET and the cursor FILTER together
** This is accomplished as a manual rewrite based on the bd:slice query pattern with the addition of a cursor/index (such as STR(?subject_var)) and ORDER BY statement using this index. A manual evaluation of the results is required - to determine the largest index value returned for the query. After the first query, each one has to include a FILTER() statement based on the largest index value returned, along with a LIMIT and OFFSET.
* For range:
** Use the query pattern as-is and &lt;i&gt;SELECT (COUNT(*) AS ?range)&lt;/i&gt;

== wikibase:decodeURI ==
wikibase:decodeUri reverses percent-encoding (for example, é → %C3%A9, comma → %2C, etc.) in a string. It is the explicit inverse of SPARQL 1.1's ENCODE_FOR_URI and turns IRIs such as Wikipedia/Commons sitelink URLs into human-readable strings. 

Basically, this is a function that is used in BINDs and projected variable clauses to create  human output. An exemplary Blazegraph query that translates a Russian Wikipedia sitelink to readable characters is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?sitelink ?decodedURI ?title WHERE {
  ?sitelink schema:about ?item ;
                schema:isPartOf &lt;https://ru.wikipedia.org/&gt; .
  BIND(wikibase:decodeUri(STR(?sitelink)) AS ?decodedURI)
 } LIMIT 10
&lt;/syntaxhighlight&gt;

The following illustrates one of the results:
* ?item: http://www.wikidata.org/entity/Q14760226 
* ?sitelink: https://ru.wikipedia.org/wiki/%D0%90%D0%B1%D1%83%D0%BB%D1%8C_%D0%A5%D0%B0%D0%B4%D0%B6%D0%B4%D0%B6%D0%B0%D0%B4%D0%B6_I
* ?decodedURI: https://ru.wikipedia.org/wiki/Абуль_Хаджджадж_I

Regrettably, there is no equivalent decodeURI functionality in SPARQL 1.1. It cannot be created since that would require iterating over a string to assemble &lt;i&gt;multi-byte UTF-8 sequences&lt;/i&gt; (as seen above, %C3%A9 becomes é). SPARQL does not support loops or byte-level operations, so a complete and correct decoder is not possible.

The only approach is to translate the URI after the SPARQL results are returned. Every programming language has a URL decode component. Using one of these languages (such as python or perl) on a Mac or Linux system, the command line instruction is:
:&lt;syntaxhighlight lang=&quot;text&quot;&gt;
python3 -c &quot;import sys,urllib.parse as u; print(u.unquote(sys.argv[1]))&quot; &quot;uri_string_to_decode&quot;
&lt;/syntaxhighlight&gt;
OR
:&lt;syntaxhighlight lang=&quot;text&quot;&gt;
perl -CS -MURI::Escape -MEncode -e 'print decode_utf8(uri_unescape($ARGV[0])),&quot;\n&quot;' &quot;uri_string_to_decode&quot;
&lt;/syntaxhighlight&gt;

On Windows, it can be accomplished in Powershell by:
:&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
pwsh -c &quot;[Uri]::UnescapeDataString('uri_string_to_decode')&quot;
&lt;/syntaxhighlight&gt;

An alternate strategy when decoding Wikipedia sitelinks is to instead retrieve the “title” from the value of the predicate, schema:name. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?title WHERE {
  ?article schema:about ?item ;
           schema:isPartOf &lt;https://ru.wikipedia.org/&gt; ;
           schema:name ?title .
} LIMIT 10 
&lt;/syntaxhighlight&gt;

Note that this does not return the full URI, but only the last path segment. So, at best, it is an approximation to aid in human readability.

=== Query Rewrite Pattern for wikibase:decodeURI ===
There is no general rewrite pattern. decodeURI() should be handled manually, at the client, using the command line instructions shown above. 

Specifically for sitelinks (identified as the subject variable of a triple with the predicate, schema:about), a decoded subset of the full URI could be obtained by retrieving the value of the variable’s schema:name predicate. 

&lt;i&gt;?something schema:about ?item … BIND(wikibase:decodeUri(STR(?something)) AS ?decodedURI)&lt;/i&gt; becomes &lt;i&gt;?something schema:about ?item ; schema:name ?decodedTitle&lt;/i&gt;. However, further string manipulations of ?decodedURI will require updating since ?decodedURI and ?decodedTitle are not equivalent. 

== Footnotes ==</text>
      <sha1>h564gjpbx0fyh2ncw7nn2o7ghyvjsij</sha1>
    </revision>
    <revision>
      <id>2435513</id>
      <parentid>2435512</parentid>
      <timestamp>2026-07-12T01:07:29Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>lang=&quot;sparql&quot;</comment>
      <origin>2435513</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="35791" sha1="c4okvwokem9y39bvmegq2r69j0uf2lb" xml:space="preserve">This Wikitech page catalogs the rewrites needed to translate several utility query services provided by Blazegraph into portable, standards-conformant SPARQL 1.1. To port these queries, each Blazegraph-specific construct must be substituted with an equivalent SPARQL 1.1 pattern, restructured into a portable form, or (where no equivalent exists) removed with an explanatory note. 

The rewrites described in this document cover six such constructs: 
* The wikibase:label service 
* Named subqueries 
* Inline query hints 
* bd:sample and bd:slice result-limiting predicates
* The wikibase:decodeURI function 
* The wikibase:label service  

For each construct, the sections that follow explain its Blazegraph-specific semantics and specify the rewrite.

== wikibase:label ==
The wikibase:label service auto-resolves an entity's label, description, and aliases into result variables, honoring a language priority list with fallback. There are two modes used to define variable names - a manual and an automatic mode.

In the manual mode, the query writer declares the variable names. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   SERVICE wikibase:label {
       bd:serviceParam wikibase:language &quot;de,en&quot; .
       ?item rdfs:label ?label .
       ?item schema:description ?description .
       ?item skos:altLabel ?alternate . }
}
&lt;/syntaxhighlight&gt;

This query produces results as shown below:
[[File:Rewrite Label-Fig 1.png|Figure 1]]

In automatic mode, no names are defined and the service projects specific names for any variables, ?x. The projected names are ?xLabel, ?xDescription, ?xAltLabel. A query could be written as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel WHERE {
   ?item wdt:P31 wd:Q515 .
   SERVICE wikibase:label { 
       bd:serviceParam wikibase:language &quot;[AUTO_LANGUAGE],de,en&quot;. }
}
&lt;/syntaxhighlight&gt;

This query produces the same results but with different names for the label, description and alias outputs.

When rewriting a query to produce equivalent results, the rewrite must consider:
* Language priority + fallback — wikibase:language is a comma-separated priority list where the first available language wins
** Usage of [AUTO_LANGUAGE] will be continued as a UI-only feature (it expands the list of languages to include the requester's UI/browser language)
* mul fallback — a label or alias under the multilingual code “mul” applies to all languages and is consulted after the listed languages
** There are no “mul” descriptions 
* Allowance for unbound labels, descriptions and aliases
* QID fallback for labels (only)
** If no label exists in any requested language or mul, the entity ID (e.g., Q42) is returned 
** There is no fallback for descriptions or altLabels (they are unbound if not found)
* skos:altLabel can be multivalued
** label service returns all aliases of the best language as one comma-joined string
* Applicable to both items and properties 

At its most basic, the rewrite must drop the service call, determine each entity variable and its related label/description/alias variable names, and then return those variable values based on SPARQL OPTIONAL clauses. 

Why SPARQL OPTIONALs? Because defining a raw triple pattern (such as ?item schema:description ?description) without OPTIONAL requires the item to have a description. If one is not defined, then the entity will be deleted from the result set. This is not the behavior of the wikibase:label service.

To rewrite the first query in this section, a naive approach (but one that illustrates what is happening “under the covers”) produces the following query:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   # --- label: priority de &gt; en &gt; mul, else QID ---
   OPTIONAL { ?item rdfs:label ?label_de  . FILTER(LANG(?label_de)  = &quot;de&quot;)  }
   OPTIONAL { ?item rdfs:label ?label_en  . FILTER(LANG(?label_en)  = &quot;en&quot;)  }
   OPTIONAL { ?item rdfs:label ?label_mul . FILTER(LANG(?label_mul) = &quot;mul&quot;) }
   BIND(COALESCE(STR(?label_de), STR(?label_en), STR(?label_mul),
                 STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?label)
   # --- description: priority de &gt; en &gt; mul, else unbound ---
   OPTIONAL { ?item schema:description ?desc_de  . FILTER(LANG(?desc_de)  = &quot;de&quot;)  }
   OPTIONAL { ?item schema:description ?desc_en  . FILTER(LANG(?desc_en)  = &quot;en&quot;)  }     
   BIND(COALESCE(STR(?desc_de), STR(?desc_en)) AS ?description)
   # --- aliases: priority de &gt; en, multi-valued → GROUP_CONCAT per language ---
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_de) WHERE {
         ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;de&quot;)
      } GROUP BY ?item
   }
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_en) WHERE {
        ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;en&quot;)
      } GROUP BY ?item
   }
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_mul) WHERE {
        ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;mul&quot;)
      } GROUP BY ?item
   }
   BIND(COALESCE(?alt_de, ?alt_en, ?alt_mul) AS ?alternate)
}
&lt;/syntaxhighlight&gt;

There are several important things to note:
* All the resulting label/description labels are COALESCE’d as STR(?x) in order to remove the language tags
** This is needed since wikibase:label does not return language tags, but plain literals
** This is not needed for the aliases since GROUP_CONCAT returns a plain literal
* “mul” must be explicitly added as a language
* The COALESCE order is defined by the language priority with “mul” as the lowest priority
* The label’s COALESCE falls back to the entity QID if there are no labels in the specified languages
* Since aliases are multi-valued, each language’s results are GROUP_CONCAT'ed in their own GROUP BY ?item subqueries, then COALESCE'd for language priority
** There is a performance implication for this - the subqueries (SELECT clauses within the OPTIONALs) are always executed first and will concatenate alias names across all items with “de” or “en” aliases 
*** This is because there are no binding constraints on ?item
** To improve the performance, the constraining triple(s) for ?item (in this case, &lt;i&gt;?item wdt:P31 wd:Q515&lt;/i&gt;) should be moved inside the sub-SELECT scope

Executing the rewritten query obtains the following (equivalent but with different order) results:
[[File:Rewrite Label-Fig 2.png|Figure 2]]

The rewritten query was characterized as “naive” since it included the COALESCE. That forces an explicit ordering of the variable binding. However, it is not needed since the OPTIONAL clause is a left join. Once the first OPTIONAL binds ?label to the German label, the same variable in the next OPTIONAL's triple pattern is no longer a fresh binding. It becomes a join constraint: ?item rdfs:label ?label must now match the already-bound German literal, and FILTER(LANG(?label) = &quot;en&quot;) on a @de literal is false. Therefore, the “en” OPTIONAL adds nothing and the German value survives. If a German label did not exist (?label is unbound), the English OPTIONAL is free to bind it; then mul; and so on. The left joins themselves implement &quot;first match wins&quot; which is what COALESCE was doing (making it redundant).

What remains is only the QID fallback for ?label and stripping the language tag. That can be written without COALESCE:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
BIND(IF(BOUND(?label), STR(?label), STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?itemLabel)
&lt;/syntaxhighlight&gt;

Given this discussion, the rewritten query becomes:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   # label: de &gt; en &gt; mul into one shared ?lbl (left joins give priority), else QID
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;de&quot;)  }
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;en&quot;)  }
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;mul&quot;) }
   BIND(IF(BOUND(?lbl), STR(?lbl), STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?label)
   # description: de &gt; en into shared ?dsc, no mul and no fallback; 
   # STR(?dsc) to remove lang tag
   OPTIONAL { ?item schema:description ?dsc . FILTER(LANG(?dsc) = &quot;de&quot;) }
   OPTIONAL { ?item schema:description ?dsc . FILTER(LANG(?dsc) = &quot;en&quot;) }
   BIND(STR(?dsc) AS ?description)
   # aliases: de &gt; en &gt; mul into shared ?alternate, constrained, GROUP_CONCAT
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;de&quot;)  
       } GROUP BY ?item }
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;en&quot;)  
       } GROUP BY ?item }
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;mul&quot;) 
       } GROUP BY ?item }
}
&lt;/syntaxhighlight&gt;

The rewrite for the second query in this section (using automatic variable name binding and AUTO-LANGUAGE) uses the same overall pattern. But, there are a few items to note:
* The AUTO_LANGUAGE UI token will be maintained - so, it can be substituted as a language directly in the rewritten query (e.g., LANG(?x) = “AUTO_LANGUAGE”)
** Outside of the UI, there is no equivalent concept - all languages must be explicitly declared in the query
* If the query projected additional entity variables beyond ?item (e.g. ?country), each would need its own blocks for determining ?countryLabel, ?countryDescription and ?countryAlias 

=== Label Service Results for an Unbound Variable ===

It is possible that the variable whose label is reported is not bound when the label results are returned. An example is a request for all plumbers' names and their mothers' names. Not all the plumbers' names are defined and many of their mothers are unknown. 

For Blazegraph, this query would be written:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt; 
SELECT ?a ?aLabel ?b ?bLabel WHERE {
  ?a wdt:P31 wd:Q5 .
  ?a wdt:P106 wd:Q252924 .
  OPTIONAL { ?a wdt:P25 ?b } 
  SERVICE wikibase:label { 
       bd:serviceParam wikibase:language &quot;en&quot;. }
}
&lt;/syntaxhighlight&gt;

Using the rewrite pattern currently defined, the rewrite would appear as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt; 
SELECT ?a ?aLabel ?b ?bLabel WHERE {
  ?a wdt:P31 wd:Q5 .
  ?a wdt:P106 wd:Q252924 .
  OPTIONAL { ?a wdt:P25 ?b } 
  OPTIONAL { ?a rdfs:label ?aLabel . FILTER ( lang(?aLabel) = 'en' ) }
  OPTIONAL { ?b rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) }
}
&lt;/syntaxhighlight&gt;

But, this query likely times out. Why? Each OPTIONAL clause is executed after &quot;pushing down&quot; all currently bound variables. The variable, ?a, is clearly bound by the triples in the second and third lines. So, the evaluation of the OPTIONAL clause for ?a's label is efficient. However, the variable, ?b, becomes bound in the &lt;i&gt;OPTIONAL { ?a wdt:P25 ?b }&lt;/i&gt; clause, but that is not visible to the clause requesting ?b's label. The clause, &lt;i&gt;OPTIONAL { ?b rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) }&lt;/i&gt;, becomes a request for all possible entities and their labels. Then, this is resolved using a left join across the OPTIONALs' results.

An approach to correct this is to check that the variable is bound before executing the OPTIONAL clause to retrieve its label. Taking this approach, the query would be rewritten as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt; 
SELECT ?a ?aLabel ?b ?bLabel WHERE {
  ?a wdt:P31 wd:Q5 .
  ?a wdt:P106 wd:Q252924 .
  OPTIONAL { ?a wdt:P25 ?b } 
  BIND(IF(BOUND(?a), ?a, &quot;unbound&quot;) AS ?al)
  OPTIONAL { ?al rdfs:label ?aLabel . FILTER ( lang(?aLabel) = 'en' ) }
  BIND(IF(BOUND(?b), ?b, &quot;unbound&quot;) AS ?bl)
  OPTIONAL { ?bl rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) }
}
&lt;/syntaxhighlight&gt;

This works as desired, returning a set of results equivalent to that from Blazegraph.

Clearly, there are multiple ways to modify the query above. 
# Since ?a is clearly bound, the IF(BOUND(?a), ...) check is not needed
# Or, the first OPTIONAL clause could be rewritten as:
::&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
OPTIONAL { ?a wdt:P25 ?b .
   OPTIONAL { ?b rdfs:label ?bLabel . FILTER ( lang(?bLabel) = 'en' ) } }
&lt;/syntaxhighlight&gt;

However, a straightforward and simple BIND(IF(BOUND(?x), ... check requires less analysis and is still performant.

=== Query Rewrite Pattern for wikibase:label ===
Collecting all the requirements and considerations above, the rewrite pattern can be explained as:
* Determine the languages to be output, splitting wikibase:language values using a comma delimiter
** And their priority order
* Determine the names of the projected variables 
** Which then defines the types of OPTIONAL “blocks” needed - label/description/alias
* Create those blocks based on the pattern shown in the last query above (BIND checking for a bound variable and then the OPTIONAL clauses requesting the label in the specified and prioritized languages), and substitute them in the query in place of the wikibase:label service
** For the &lt;i&gt;alias&lt;/i&gt; labels, taking care to re-insert the constraining triple(s) for the variable(s) in the OPTIONAL clauses' SELECT subqueries' body 
*** This is needed since those SELECT subqueries are executed first and their results used in the OPTIONAL clauses left-join operation at the end of processing

== Named Subqueries ==
Although SPARQL supports nested subqueries, it does not support assigning them a name or specifically “including” them at specific (sometimes multiple) points in a scoping query. Blazegraph allows this using the keywords, INCLUDE, AS and WITH. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer
WITH {
  SELECT ?person WHERE {
    ?person wdt:P31 wd:Q5 ; wdt:P106 wd:Q901 .   # humans who are scientists
  }
} AS %scientists
WHERE {
  INCLUDE %scientists .
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

The query first executes the &lt;i&gt;WITH { … } AS %name&lt;/i&gt; subquery, and does this only once. Its results are materialized into a named temporary set, and every &lt;i&gt;INCLUDE %name&lt;/i&gt; joins against that same cached set. It's effectively a SQL Common Table Expression (CTE) using WITH … AS. 

Named subqueries serve three purposes: 
* Clarity - cleanly defining the query
* Reuse - allowing an expensive subquery to be executed once and used in several places 
* Join-order control - forcing the subquery to run first, rather than letting the query optimizer to insert it

Unfortunately, SPARQL 1.1 has no equivalent to named subqueries. The portable equivalent is an inline sub-SELECT that duplicates the code of the WITH clause and replaces each &lt;i&gt;INCLUDE %name&lt;/i&gt; with the body of the sub-SELECT. For the example above:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer WHERE {
  { SELECT ?person WHERE {
      ?person wdt:P31 wd:Q5 ; wdt:P106 wd:Q901 .
  } }
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

The most important question is whether the subqueries are simply copied wherever they are referenced. And, for ordinary, deterministic queries, the answer is yes. The sub-SELECT body can be pasted at each INCLUDE site&lt;ref&gt;This is actually a worst case rewrite since the pattern can be simplified to only paste within each unique query scope. This is discussed in more detail in the Query Rewrite sub-section.&lt;/ref&gt;.

Again, unfortunately, &quot;simply copied&quot; hides three problems:  
* Complexity, when a sub-query is copied multiple times and the query text “explodes” 
* Performance, since the compute-once guarantee is lost
** Blazegraph evaluates the block once regardless of the number of references to it
** Instead, if it is inlined at N sites, then the query is evaluated N times
** For a “cheap” subquery, this is mainly irrelevant
** For an “expensive” subquery referenced several times, a large performance cost could be incurred
* Correctness, when the sub-query is non-deterministic or mints blank nodes AND occurs multiple times
** Non-deterministic queries include functions such as RAND() and UUID()/STRUUID()
*** When Blazegraph executes the sub-query once, every INCLUDE sees the same bound variables and values
*** Instead, if inlined multiple times, each copy re-evaluates those functions and returns different values
** The sub-query may return variables as blank nodes
*** As above, when Blazegraph executes the sub-query once, every INCLUDE sees the same bound variables and bnode values
*** Instead, if inlined multiple times, each copy returns different bnode identities
** Other examples of non-deterministic results are subqueries that use:
*** A LIMIT but no ORDER BY, making the result selection defined at execution time
*** GROUP_CONCAT, since the SPARQL aggregate functions operate over unordered sets
*** SAMPLE()
*** A federated service call to an external endpoint
**** If there are N separate calls to the federated service, results could vary between the calls due to changes in the data, a LIMIT on the federated result, or other non-deterministic behavior at the endpoint
*** Blazegraph-specific sampling utilities such as bd:slice and bd:sample

If non-deterministic or blank node results are included in the solution set, &lt;i&gt;AND the named subquery is used more than once&lt;/i&gt;, it cannot just be copied but must first be executed/materialized. Then, its results can be manually added back into the query as a VALUES block.

For example, rewriting the scientist query above using the VALUES approach (even though this is deterministic), results in the following: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer WHERE {
  VALUES ?person { wd:Q4118502 wd:Q100156581 … }   # Results of running %scientists once
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

There are a few obvious problems with the VALUES approach. First, it is a two-step manual or programmatic process. Second, it is cumbersome when the result set is large (thousands of results).

=== Query Rewrite Pattern for Named Subqueries ===
For rewriting a named subquery, its contents (within the WITH { … } AS clause) are replaced by an inlined &lt;i&gt;{ SELECT variables WHERE { … } }&lt;/i&gt; clause. If there is only a single reference to the sub-query, the result of inlining is equivalent to the original query. 

However, questions arise when there are multiple references to the sub-query and these occur in multiple scopes&lt;ref&gt;Examples of different scopes in a SPARQL query are having nested sub-queries or separate UNION branches.&lt;/ref&gt;. One solution to this is to rewrite the sub-query in each scope. Further complications then arise only when the results of the sub-query are non-deterministic and this is visible in the externalized / observable result variables. 

There is complexity in determining these two characteristics (non-deterministic and projected). To determine them, the query’s algebra tree&lt;ref&gt;Obtained using Jena ARQ or rdflib libraries&lt;/ref&gt; must be analyzed. This can be used to first define whether multiple scopes are involved. If the enclosing graph pattern for each reference to a named subquery is the same - then there is only one scope (and no rewrite problems). However, if two INCLUDEs have different enclosing patterns, then their potential results need to be examined. 

A simple way to determine if the results are non-deterministic due to generated blank nodes is to actually run the sub-query and see if any results are reported with type = “bnode”&lt;ref&gt;It is important to distinguish Wikidata skolemized blank nodes, and query engine-minted nodes. Wikidata’s blank nodes will be returned in a query binding as type=&quot;uri&quot; with a specific, consistent IRI value.&lt;/ref&gt;. If so and the subquery is referenced in multiple scopes, then that query is not eligible for rewriting.

Other simple checks for non-deterministic results is whether the subquery includes bd:slice or bd:sample (which have approximate rewrites), or a LIMIT without an ORDER BY. 

Lastly, if the subquery uses SAMPLE(), RAND(), UUID(), STRUUID() or GROUP_CONCAT(), and if that use occurs in multiple scopes, and if the results of the function calls are observable in the results, then the query is &lt;b&gt;ineligible&lt;/b&gt; for rewrite. Being “observable” occurs if the functions appear in the SELECT clause, affect which result sets are returned (used in FILTER/HAVING or ORDER BY under a LIMIT/OFFSET), or feed a projected BIND. 

Examples of these are:
* A SELECT variable (for example, SELECT ?k (SAMPLE(?v) AS ?s))
* A FILTER/HAVING clause (e.g., HAVING(SAMPLE(?v) &gt; 5))
* An ORDER BY (SAMPLE(?v)) ... LIMIT n clause (since the results which “clear” the LIMIT are dependent on the SAMPLE selection) 

=== What about QLever’s Materialized Views? ===
QLever has defined a feature (still in beta at the time of writing) called [https://docs.qlever.dev/materialized-views/ materialized views] which seems similar to Blazegraph’s named subqueries. Both provide the capability to create a named, compute-once result set that is referenced by name. However, there are several differences and explicit issues. The most important of these are: 
* SPARQL 1.1 compliance:
** Neither Blazegraph named subqueries nor QLever materialized views are SPARQL 1.1 compliant
** Also, the creation of a materialized view requires server-level access (not allowed on WDQS)
* Definition location: 
** A named subquery (WITH … AS %x) is defined inside the query itself - it is self-contained and recomputed every run
** A materialized view is created out of band and stored on disk (dependent on server-side state) 
* Lifetime and freshness:
** Since named subqueries are recomputed on every query execution, they are never stale
** Materialized views are persistent and must be recreated if they go stale 

== Query Hints ==
QLever does not expose a Blazegraph-style query-hint system and this is not a part of the SPARQL 1.1 standard. So, the rewrite pattern for Blazegraph hint:* directives is to strip them entirely.  

It is important to note that QLever's optimizer can be influenced by the structure of a SPARQL query. It is recommended to use subqueries to force early evaluation of a portion of a query. When enclosing a block inside a SPARQL subquery (SELECT ... WHERE { ... }), QLever will execute that block before joining its results with the remainder of the query.

== bd:sample ==
bd:sample is a Blazegraph service that selects a bounded subset of solutions from a graph pattern &lt;i&gt;without&lt;/i&gt; fully materializing and sorting the entire set of results. Its purpose is to keep otherwise-expensive queries under the WDQS timeout by working on a sample instead of the full data. 

For example, the following query returns 2000 “locations” (&lt;i&gt;wdt:P276 ?value&lt;/i&gt;) and orders them by the number of distinct ?items that reference them. (Note however, that this is the ordering within the sample set of results and not across Wikidata.)

&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?value (COUNT(DISTINCT ?item) as ?count) WHERE {
 SERVICE bd:sample {
   ?item wdt:P276 ?value .
   bd:serviceParam bd:sample.limit 2000 . }
} GROUP BY ?value
  ORDER BY DESC(COUNT(DISTINCT ?item))
&lt;/syntaxhighlight&gt;

The query pattern defined within the service binds the relevant variables (in this case, ?item and ?value). These can then be used in the surrounding SPARQL.

The possible service parameters are:
{| class=&quot;wikitable&quot;
|-
! Parameter !! Usage !! Default
|-
| bd:sample.limit || Maximum number of result sets to return || 100
|-
| bd:sample.sampleType || “RANDOM”, “EVEN” (evenly spaced sample), “DENSE” (first N) || RANDOM
|-
| bd:sample.seed || RANDOM's seed || 0 
|}

Rewriting the service depends on the sampleType parameter, but none of the rewrites preserve the performance characteristics of bd:sample. For all rewrites, the engine must evaluate the inner pattern first and run to completion. 

The closest SPARQL 1.1 query to that shown above is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?value (COUNT(DISTINCT ?item) AS ?count) WHERE {
  { # Translate the query pattern within the service to a SELECT query
    SELECT ?item ?value WHERE {    
      ?item wdt:P276 ?value .
    } ORDER BY RAND()      # Default is RANDOM, so apply that ordering
      LIMIT 2000           # LIMIT 2000
  }
} GROUP BY ?value
  ORDER BY DESC(COUNT(DISTINCT ?item))
&lt;/syntaxhighlight&gt;

The above query does complete on QLever, but a query that returns a larger result set (for example, asking for a subset of humans, Q5) or is more complex could exceed the 60 second timeout. In all cases, applying a LIMIT to the inner query is needed to force a cut-off. That is also why bd:sample has a limit parameter.

It is valuable to note that the SPARQL 1.1 query could be changed to remove the ORDER BY RAND() statement and just use LIMIT. That would definitely improve the performance, and would mimic choosing the “DENSE” sampleType of bd:sample. 

“EVEN” is the last sampleType value that could be used. To replicate the behavior requires positional/windowing logic that standard SPARQL 1.1 cannot express. However, the result can be approximated with the RANDOM rewrite above (a uniform random sample is usually an acceptable stand-in for an evenly-spaced one).

There is another service parameter that we did not discuss - seed. This would be used for reproducibility. Here is a Blazegraph query using that parameter:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  SERVICE bd:sample {
    ?item wdt:P31 wd:Q5 .                         # ?item is an instance of human
    bd:serviceParam bd:sample.limit 1000 .
    bd:serviceParam bd:sample.sampleType &quot;RANDOM&quot; .
    bd:serviceParam bd:sample.seed 42 .           # fixed seed → same 1000 every run
  }
}
&lt;/syntaxhighlight&gt;

The rewrite is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  { SELECT ?item WHERE {
      ?item wdt:P31 wd:Q5 .
    } ORDER BY MD5(CONCAT(&quot;42&quot;, STR(?item)))   # &quot;42&quot; plays the role of bd:sample.seed
      LIMIT 1000
               }
}
&lt;/syntaxhighlight&gt;

Note that the above query can successfully execute on a QLever server in under 60 seconds. So, simply having a large intermediate result set does not necessarily equate to query timeout.

=== Query Rewrite Pattern for bd:sample ===
These steps summarize the rewrite patterns for bd:sample:
* Lift the query &lt;pattern&gt; out of SERVICE bd:sample {…}  and rewrite it as an inner SELECT … WHERE { &lt;pattern&gt; }, projecting the variables present in the outer query 
* Append LIMIT N (the bd:sample.limit or use the default, 100, if it was omitted)
* Add an ORDER BY if the sampleType is “RANDOM” or “EVEN” 
** RAND() if a seed is not specified
** MD5(CONCAT(seed, STR(?key))) otherwise
* Leave the surrounding query (GROUP BY, COUNT, etc.) unchanged

== bd:slice ==
bd:slice is a Blazegraph service that provides efficient, stable pagination over a single triple pattern. It walks through a very large result set in deterministic chunks without re-sorting on each request. Its purpose is to keep otherwise-expensive queries under the WDQS timeout by working on a subset instead of the full data. 

The most “mechanically” equivalent SPARQL 1.1 pattern is to perform the query with an ORDER BY, and OFFSET / LIMIT. However, this becomes increasingly non-performant as the OFFSET size grows - since OFFSET skips rows in the result set by first generating and then discarding them. 

Another way to rewrite the query using SPARQL 1.1 is to create an “index” and use that to segment the results. Taking this approach each page carries a similar performance cost. 

For example, the following Blazegraph query is used to return result sets of 10000 humans. Subsequent calls increase the offset by 10000 and add 10000 to the limit. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
 SERVICE bd:slice {
   ?item wdt:P31 wd:Q5 .                    # single triple pattern (required)
   bd:serviceParam bd:slice.offset 0 .
   bd:serviceParam bd:slice.limit 10000 .
 }
}
&lt;/syntaxhighlight&gt;

To rewrite this query using SPARQL 1.1 and a query-relevant index requires some manual construction … The largest value of the value from a run is inserted into a FILTER on the index.

For the query above, the first iteration is written as: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .             
} ORDER BY (str(?item))
  LIMIT 10000  
&lt;/syntaxhighlight&gt;

The second iteration becomes: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .
  FILTER(STR(?item) &gt; &quot;http://www.wikidata.org/entity/Q100310806&quot;)
} ORDER BY STR(?item)
  LIMIT 10000 
&lt;/syntaxhighlight&gt;

Where the entity, Q100310806, is the “last” ?item output by the previous iteration.

It is important to note that because of the indexes and performance characteristics of QLever, it is also possible to execute the query unconstrained  by order or limits. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .
}  
&lt;/syntaxhighlight&gt;

A total of 13M+ results are output, and indicates that bd:slice was a performance optimization required by Blazegraph as opposed to addressing a query-related need.

As for bd:sample, there is another service parameter for bd:slice that should be discussed - range. In range mode, bd:slice doesn't return rows of results, but returns a single value - a count of the solutions that is computed from index boundaries. A variable is named and bd:slice.range estimates a count. Note that word, estimates. This is problematic since it is usually an overestimate and done as another performance optimization.

Consider these two Blazegraph queries:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?range WHERE {
  SERVICE bd:slice {
    ?item wdt:P569 ?dob .                       # single triple pattern
    bd:serviceParam bd:slice.range ?range .     # ?range ← fast index count of matches
  }
}
&lt;/syntaxhighlight&gt;

And:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?range WHERE {
  SERVICE bd:slice {
    ?item wdt:P569 ?dob .
    FILTER(?dob &gt;= &quot;1900-01-01&quot;^^xsd:dateTime &amp;&amp; ?dob &lt; &quot;2000-01-01&quot;^^xsd:dateTime)
    bd:serviceParam bd:slice.range ?range .
  }
}
&lt;/syntaxhighlight&gt;

Both of these queries return a ?range of approximately 8074600 (varying +/- 3).

However, a corresponding query using SPARQL 1.1, gives exact results.
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT (COUNT(*) AS ?range) WHERE {
  ?item wdt:P569 ?dob .
  FILTER(?dob &gt;= &quot;1900-01-01T00:00:00&quot;^^xsd:dateTime &amp;&amp; 
               ?dob &lt; &quot;2000-01-01T00:00:00&quot;^^xsd:dateTime)
}
&lt;/syntaxhighlight&gt;

Which returns a count of 5457644. With the FILTER removed, the count is 8074572 (pretty close to the overall Blazegraph estimate!).

=== Query Rewrite Pattern for bd:slice ===
The following list defines the rewrite patterns for bd:slice with limit and offset, or with range:
* For limit/offset:
** Evaluate first if limit/offset are actually needed. For many queries, QLever’s indexes and performance are sufficient to remove the need for this optimization.
** If the query does timeout, then pagination is needed. The simplest portable form is to use ORDER BY &lt;key&gt; + OFFSET + LIMIT, bumping OFFSET with each query request. But, note that OFFSET cost grows with depth. 
*** For deep paging, it is recommended to use keyset pagination. Pick a deterministic sort key (e.g. STR(?subject)), ORDER BY it, issue an initial LIMIT query, then for each subsequent page add &lt;i&gt;FILTER(STR(?subject) &gt; “last_value_from_previous_query”)&lt;/i&gt; with the same ORDER BY and LIMIT. 
*** Keyset replaces the growing OFFSET with an index seek
*** Remember - do not use both OFFSET and the cursor FILTER together
** This is accomplished as a manual rewrite based on the bd:slice query pattern with the addition of a cursor/index (such as STR(?subject_var)) and ORDER BY statement using this index. A manual evaluation of the results is required - to determine the largest index value returned for the query. After the first query, each one has to include a FILTER() statement based on the largest index value returned, along with a LIMIT and OFFSET.
* For range:
** Use the query pattern as-is and &lt;i&gt;SELECT (COUNT(*) AS ?range)&lt;/i&gt;

== wikibase:decodeURI ==
wikibase:decodeUri reverses percent-encoding (for example, é → %C3%A9, comma → %2C, etc.) in a string. It is the explicit inverse of SPARQL 1.1's ENCODE_FOR_URI and turns IRIs such as Wikipedia/Commons sitelink URLs into human-readable strings. 

Basically, this is a function that is used in BINDs and projected variable clauses to create  human output. An exemplary Blazegraph query that translates a Russian Wikipedia sitelink to readable characters is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?sitelink ?decodedURI ?title WHERE {
  ?sitelink schema:about ?item ;
                schema:isPartOf &lt;https://ru.wikipedia.org/&gt; .
  BIND(wikibase:decodeUri(STR(?sitelink)) AS ?decodedURI)
 } LIMIT 10
&lt;/syntaxhighlight&gt;

The following illustrates one of the results:
* ?item: http://www.wikidata.org/entity/Q14760226 
* ?sitelink: https://ru.wikipedia.org/wiki/%D0%90%D0%B1%D1%83%D0%BB%D1%8C_%D0%A5%D0%B0%D0%B4%D0%B6%D0%B4%D0%B6%D0%B0%D0%B4%D0%B6_I
* ?decodedURI: https://ru.wikipedia.org/wiki/Абуль_Хаджджадж_I

Regrettably, there is no equivalent decodeURI functionality in SPARQL 1.1. It cannot be created since that would require iterating over a string to assemble &lt;i&gt;multi-byte UTF-8 sequences&lt;/i&gt; (as seen above, %C3%A9 becomes é). SPARQL does not support loops or byte-level operations, so a complete and correct decoder is not possible.

The only approach is to translate the URI after the SPARQL results are returned. Every programming language has a URL decode component. Using one of these languages (such as python or perl) on a Mac or Linux system, the command line instruction is:
:&lt;syntaxhighlight lang=&quot;text&quot;&gt;
python3 -c &quot;import sys,urllib.parse as u; print(u.unquote(sys.argv[1]))&quot; &quot;uri_string_to_decode&quot;
&lt;/syntaxhighlight&gt;
OR
:&lt;syntaxhighlight lang=&quot;text&quot;&gt;
perl -CS -MURI::Escape -MEncode -e 'print decode_utf8(uri_unescape($ARGV[0])),&quot;\n&quot;' &quot;uri_string_to_decode&quot;
&lt;/syntaxhighlight&gt;

On Windows, it can be accomplished in Powershell by:
:&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
pwsh -c &quot;[Uri]::UnescapeDataString('uri_string_to_decode')&quot;
&lt;/syntaxhighlight&gt;

An alternate strategy when decoding Wikipedia sitelinks is to instead retrieve the “title” from the value of the predicate, schema:name. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?title WHERE {
  ?article schema:about ?item ;
           schema:isPartOf &lt;https://ru.wikipedia.org/&gt; ;
           schema:name ?title .
} LIMIT 10 
&lt;/syntaxhighlight&gt;

Note that this does not return the full URI, but only the last path segment. So, at best, it is an approximation to aid in human readability.

=== Query Rewrite Pattern for wikibase:decodeURI ===
There is no general rewrite pattern. decodeURI() should be handled manually, at the client, using the command line instructions shown above. 

Specifically for sitelinks (identified as the subject variable of a triple with the predicate, schema:about), a decoded subset of the full URI could be obtained by retrieving the value of the variable’s schema:name predicate. 

&lt;i&gt;?something schema:about ?item … BIND(wikibase:decodeUri(STR(?something)) AS ?decodedURI)&lt;/i&gt; becomes &lt;i&gt;?something schema:about ?item ; schema:name ?decodedTitle&lt;/i&gt;. However, further string manipulations of ?decodedURI will require updating since ?decodedURI and ?decodedTitle are not equivalent. 

== Footnotes ==</text>
      <sha1>c4okvwokem9y39bvmegq2r69j0uf2lb</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:WikiDataAuthorityControlTool</title>
    <ns>116</ns>
    <id>460408</id>
    <revision>
      <id>2435514</id>
      <parentid>2435239</parentid>
      <timestamp>2026-07-12T01:10:15Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>rm dupe image size</comment>
      <origin>2435514</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6756" sha1="ejhepzbmkbsqb9ngofq6ah0hlkyw3j6" xml:space="preserve">{{Tool
| name = wd-act
| title = WikiData Authority Control Tool
| description = A tool to help WikiData editors research Authority Control data about WikiData items. Displays completeness information about an entity's Authority Control data fields, and for any missing fields, provides helpful URLs to search for missing data.
| website = https://wd-act.toolforge.org/
| keywords = authority control, WikiData
| author = Tekneos
| username = Ancient9983
| maintainer = 
| repository = https://gitlab.wikimedia.org/toolforge-repos/wd-act
| license = AGPL-3.0
| image = WikiData Authority Control Tool initial screenshot.png
| imagesize = 350
| phabricator = Tool-wd-act
| bugzilla = 
| issues = 
| sal = 
| icon = 
}}

== WikiData Authority Control Tool ==

This tool helps you research Authority Control data for WikiData Items. It displays data about common fields in the [[:en:Template:Authority_control|Authority Control template]], showing in an easy-to-use interface which fields have data, and which data is missing. For any fields which are missing data, it provides a handy link to search for the WikiData Item in the external database.

'''Project status:''' actively maintained__TOC__

=== Basic functionality ===

WikiData Authority Control Tool is intended for use by WikiData/general wiki editors. It is meant to help editors research authority control data on different entities.

As an example: let's say you're a Wikipedia editor and want to know how your favorite author or band is represented in common databases, and improve the data. This tool quickly gives you  an idea of how &quot;complete&quot; their coverage is in common external authority control sources. And anywhere where data is missing, you can get a quick jump-start towards finding missing data.

For more details about the technical implementation, see [[#Developer_documentation|the developer documentation]].

=== What WikiData Authority Control Tool does not do ===

This tool does not currently allow editing WikiData items. At the moment, it is purely intended to be used as a research convenience tool, to make it easier to find Authority Control information. It's expected that users will take the information they find and update WikiData separately.

== User documentation ==

=== Prerequisites ===
Before using this tool, you should be familiar with:
* Basic understanding of how WikiData and WikiData Items work
* The idea of Authority Control (see: https://en.wikipedia.org/wiki/Authority_control)
* Ideally, have an understanding of how Authority Control data in WikiData propogates into Wikipedia, for example, via the [[:en:Template:Authority_control|Authority control template]].

You should have:
* A web browser
* A particular WikiData Item you want to research

=== How to use WikiData Authority Control Tool ===

==== Lookup an example WikiData item ====
# Open the Authority Control Tool UI at https://wd-act.toolforge.org/
# Click the &quot;Help&quot; page
# Click one of the example WikiData items to view Authority Control data for it

==== Lookup Authority Control data on a specific WikiData item ====
# Search for the item on WikiData and get its QID
## For example, for [[d:Q169566|H. P. Lovecraft]], the QID is Q169566.
# Append the QID to the end of the Authority Control Tool's URL.
## In this example, the URL would be: https://wd-act.toolforge.org/Q169566
# Review the item's authority control data
# Optionally, if any data is missing, click the &quot;Search&quot; button to attempt to find the data in the external system.

=== How to get help and report issues ===
* Report issues at Phabricator [[phab:project/profile/8856/|here]]

== Developer documentation ==

=== OAuth Flow use ===
I'm currently planning to use the Client Credentials flow for this application. Why?

All this application does is read data from the WikiData API and display it to the user. It does not need access to any user-specific data. Therefore, it doesn't make sense to get user-specific tokens to use when calling the WikiData REST API.

If I find out in the future that something needs to change here, I can revisit.

==== OAuth Reference info ====

* &lt;nowiki&gt;https://www.mediawiki.org/wiki/OAuth/For_Developers&lt;/nowiki&gt;
* &lt;nowiki&gt;https://www.wikidata.org/wiki/Wikidata:REST_API&lt;/nowiki&gt;

=== Technical implementation ===
==== Libraries/frameworks used ====
These libraries/tools/frameworks are currently used in this project:

* Bootstrap
** UI framework/components
* Flask
** Web server framework

* Python
** programming language any server-side logic is written in
* Python 'requests' library
** Used for making HTTP requests in Python
* wikibase-rest-api-client Python library (&lt;nowiki&gt;https://pypi.org/project/wikibase-rest-api-client/&lt;/nowiki&gt;)
** Used to simplify calling the [[d:Wikidata:REST_API|WikiData REST API]] and parse the API response data into usable Python objects
* Python.gitignore from &lt;nowiki&gt;https://github.com/github/gitignore&lt;/nowiki&gt;

=== Local development ===

==== Set up ====

How to set up development environment:
# Clone the repository from GitLab here: https://gitlab.wikimedia.org/toolforge-repos/wd-act
# Open the VS Code Workspace in the root of the repository in VS Code.

==== Configure ====

How to configure:
# Update run.sh or run.ps1 with the client id/secret of the API key. The software uses this credential to retrieve data from the WikiData REST API.

==== Build and test ====

How to build and run locally:
# There's no build step since we're working in Python.
# To run the project, open a terminal in the root of the repository. Then run the 'run.ps1' script in PowerShell, or the 'run.sh' script in Bash to run the web app.
# Open a browser to localhost:5000 to browse the web app.

=== How to contribute ===
The WikiData Authority Control Tool maintainers welcome contributions!
* Bug reports/feature requests
* Pull requests

==== Contribution process ====
# Bug reports/feature requests:
## First, setup an account in Phabricator to submit issues.
## Navigate to the WikiData Authority Control Tool project in Phabricator: https://phabricator.wikimedia.org/project/view/8856/
## Create a bug report/feature request there

# Pull requests
## Navigate to the project's GitLab repo here: https://gitlab.wikimedia.org/toolforge-repos/wd-act
## Clone the repo and make a branch to work on your changes locally
## Make your changes, test and validate
## Create a pull request in GitLab, with a request to merge into the main branch

== Credits ==

List any contributors and individuals, teams, or organizations who contributed to your development of this tool. 

* [[User:Ancient9983]]

== License ==

GNU Affero General Public License, v3.

Link to license: https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/blob/main/LICENSE.txt</text>
      <sha1>ejhepzbmkbsqb9ngofq6ah0hlkyw3j6</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:Deadrat</title>
    <ns>3</ns>
    <id>460441</id>
    <revision>
      <id>2435500</id>
      <timestamp>2026-07-11T18:12:36Z</timestamp>
      <contributor>
        <username>StrikerBot</username>
        <id>8475</id>
      </contributor>
      <comment>Welcome to Toolforge!</comment>
      <origin>2435500</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="619" sha1="qh8i53hujmpdi992jqhprcujuayo67o" xml:space="preserve">== Welcome to Toolforge! ==
Hello Deadrat, welcome to the Toolforge project!  Your request for access was processed, and you should be able to use ssh to connect to &lt;tt&gt;login.toolforge.org&lt;/tt&gt; and create and manage tools at https://toolsadmin.wikimedia.org/.

Check the [[Help:Toolforge|Toolforge help page]] for tips on using your account.  You can also ask questions in our IRC channel at {{irc|wikimedia-cloud}} or send an e-mail to our mailing list &lt;tt&gt;cloud@lists.wikimedia.org&lt;/tt&gt;.

Thank you, and have fun making Tools! --[[User:StrikerBot|StrikerBot]] ([[User talk:StrikerBot|talk]]) 18:12, 11 July 2026 (UTC)</text>
      <sha1>qh8i53hujmpdi992jqhprcujuayo67o</sha1>
    </revision>
  </page>
</mediawiki>