Server Admin Log

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.47.0-wmf.1</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Server Admin Log</title>
    <ns>0</ns>
    <id>7919</id>
    <revision>
      <id>2412193</id>
      <parentid>2412188</parentid>
      <timestamp>2026-05-10T02:00:19Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image</comment>
      <origin>2412193</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="311656" sha1="14ic4mzfvr5dd5d5pdcmz9235u7krsi" xml:space="preserve">
== 2026-05-10 ==
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-05-09 ==
* 10:34 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix dsl column size - oblivian@cumin1003&quot;
* 10:34 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix dsl column size - oblivian@cumin1003
* 10:33 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix dsl column size - oblivian@cumin1003
* 10:33 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix dsl column size - oblivian@cumin1003&quot;
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s)
* 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:06 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1267.eqiad.wmnet with OS bookworm
* 01:06 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 01:06 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 00:49 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1267.eqiad.wmnet with reason: host reimage
* 00:44 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1267.eqiad.wmnet with reason: host reimage
* 00:29 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1267.eqiad.wmnet with OS bookworm
* 00:17 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1267.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED

== 2026-05-08 ==
* 23:55 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1267.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:35 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1267
* 23:32 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1267
* 23:30 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 23:30 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1267] - vriley@cumin1003&quot;
* 23:30 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1267] - vriley@cumin1003&quot;
* 23:26 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 23:22 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1266.eqiad.wmnet with OS bookworm
* 23:22 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 23:11 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 22:54 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1266.eqiad.wmnet with reason: host reimage
* 22:46 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1266.eqiad.wmnet with reason: host reimage
* 22:26 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1266.eqiad.wmnet with OS bookworm
* 22:16 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1266.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:56 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1266.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:55 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1266
* 21:53 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1266
* 21:52 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:52 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1266] - vriley@cumin1003&quot;
* 21:51 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1266] - vriley@cumin1003&quot;
* 21:45 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 21:42 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS bookworm
* 21:42 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 21:41 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 21:24 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage
* 21:19 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage
* 20:54 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS bookworm
* 20:44 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1265.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:32 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1265.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:31 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1265
* 20:30 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1265
* 20:29 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:29 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1265] - vriley@cumin1003&quot;
* 20:29 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1265] - vriley@cumin1003&quot;
* 20:24 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 20:01 ryankemper: [WDQS] Added several more requestctl rules. They've helped marginally, but not enough to restore the service. Unless we find an obvious smoking gun, expect noise to continue for the timebeing :/
* 19:42 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 19:41 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 19:41 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 19:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 18:07 ryankemper: [WDQS] After those 2 requestctl rules, requests went down 20%, error rate decreased significantly, p50 cut almost in half, but the service is still unstable, likely we'll need to identify more throttle-candidates to restore full health
* 17:53 ryankemper: [WDQS] Deployed 2 new requestctl rules; we'll see if it helps
* 16:51 topranks: enable bfd on system0.0 sub-interface ssw1-d1-eqiad
* 15:45 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on ms-backup1003.eqiad.wmnet with reason: restart
* 15:37 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup[1006,1017-1018].eqiad.wmnet with reason: restart
* 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-jumbo1001.eqiad.wmnet
* 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-jumbo1001.eqiad.wmnet
* 14:07 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:51 btullis: re-pooled wdqs-main in eqiad for [[phab:T425758|T425758]]
* 10:50 btullis@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-main,name=eqiad
* 10:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:15 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:14 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup1007.eqiad.wmnet with reason: restart
* 10:12 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:11 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:11 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:09 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:09 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:44 btullis: depooled wdqs-main in eqiad for [[phab:T425758|T425758]]
* 09:41 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 09:41 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:41 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:41 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:40 btullis@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=wdqs-main,name=eqiad
* 09:36 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 09:36 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:36 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:35 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92437 and previous config saved to /var/cache/conftool/dbconfig/20260508-093251-fceratto.json
* 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P92435 and previous config saved to /var/cache/conftool/dbconfig/20260508-092243-fceratto.json
* 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P92434 and previous config saved to /var/cache/conftool/dbconfig/20260508-091238-fceratto.json
* 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92433 and previous config saved to /var/cache/conftool/dbconfig/20260508-090230-fceratto.json
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92432 and previous config saved to /var/cache/conftool/dbconfig/20260508-085217-fceratto.json
* 08:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
* 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92431 and previous config saved to /var/cache/conftool/dbconfig/20260508-085018-fceratto.json
* 08:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92430 and previous config saved to /var/cache/conftool/dbconfig/20260508-084010-fceratto.json
* 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92429 and previous config saved to /var/cache/conftool/dbconfig/20260508-083003-fceratto.json
* 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92428 and previous config saved to /var/cache/conftool/dbconfig/20260508-081954-fceratto.json
* 08:18 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 08:17 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92427 and previous config saved to /var/cache/conftool/dbconfig/20260508-080438-fceratto.json
* 08:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 07:56 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts install5003.wikimedia.org
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: install5003.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: install5003.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 07:09 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2159: after reimage to trixie
* 06:57 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts install5003.wikimedia.org
* 06:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2159: after reimage to trixie
* 06:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2159.codfw.wmnet with OS trixie
* 06:11 moritzm: installing postorius security updates
* 05:51 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2159.codfw.wmnet with reason: host reimage
* 05:46 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2159.codfw.wmnet with reason: host reimage
* 05:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2159.codfw.wmnet with OS trixie
* 05:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2159: Reimage to Trixie
* 05:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2159: Reimage to Trixie
* 05:25 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2159.codfw.wmnet with reason: Reimage to Trixie
* 03:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1024.eqiad.wmnet with OS trixie
* 03:04 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 03:04 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 02:50 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1024.eqiad.wmnet with reason: host reimage
* 02:45 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1024.eqiad.wmnet with reason: host reimage
* 02:32 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1024.eqiad.wmnet with OS trixie
* 02:23 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1024.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 02:07 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1024.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 02:07 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1024
* 02:05 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1024
* 02:04 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 02:04 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1024] - vriley@cumin1003&quot;
* 02:04 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1024] - vriley@cumin1003&quot;
* 02:01 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 01:52 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1023.eqiad.wmnet with OS trixie
* 01:52 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 01:30 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 01:15 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1023.eqiad.wmnet with reason: host reimage
* 01:11 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1023.eqiad.wmnet with reason: host reimage
* 00:59 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1023.eqiad.wmnet with OS trixie
* 00:49 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1023.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:37 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1023.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:37 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1023
* 00:36 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1023
* 00:27 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 00:27 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1023] - vriley@cumin1003&quot;
* 00:27 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1023] - vriley@cumin1003&quot;
* 00:20 vriley@cumin1003: START - Cookbook sre.dns.netbox

== 2026-05-07 ==
* 23:25 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1022.eqiad.wmnet with OS trixie
* 23:25 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 23:24 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 23:09 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1022.eqiad.wmnet with reason: host reimage
* 23:05 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1022.eqiad.wmnet with reason: host reimage
* 22:53 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1022.eqiad.wmnet with OS trixie
* 22:25 amastilovic@deploy1003: Finished deploy [analytics/refinery@b38efb1] (thin): Regular analytics weekly train THIN [analytics/refinery@b38efb19] (duration: 01m 53s)
* 22:23 amastilovic@deploy1003: Started deploy [analytics/refinery@b38efb1] (thin): Regular analytics weekly train THIN [analytics/refinery@b38efb19]
* 22:23 amastilovic@deploy1003: Finished deploy [analytics/refinery@b38efb1]: Regular analytics weekly train [analytics/refinery@b38efb19] (duration: 03m 52s)
* 22:19 amastilovic@deploy1003: Started deploy [analytics/refinery@b38efb1]: Regular analytics weekly train [analytics/refinery@b38efb19]
* 22:18 amastilovic@deploy1003: Finished deploy [analytics/refinery@b38efb1] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@b38efb19] (duration: 01m 55s)
* 22:16 amastilovic@deploy1003: Started deploy [analytics/refinery@b38efb1] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@b38efb19]
* {{safesubst:SAL entry|1=21:27 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284828{{!}}Upgrading webonyx/graphql-php (v15.31.5 =&gt; v15.32.3)]], [[gerrit:1284834{{!}}composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T425731)}}
* 21:23 cscott@deploy1003: cscott: Continuing with deployment
* 21:17 cscott@deploy1003: cscott: Backport for [[gerrit:1284828{{!}}Upgrading webonyx/graphql-php (v15.31.5 =&gt; v15.32.3)]], [[gerrit:1284834{{!}}composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T425731)]] synced to the t
* {{safesubst:SAL entry|1=21:16 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1284828{{!}}Upgrading webonyx/graphql-php (v15.31.5 =&gt; v15.32.3)]], [[gerrit:1284834{{!}}composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T425731)]}}
* 20:53 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1021.eqiad.wmnet with OS trixie
* 20:53 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 20:49 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284575{{!}}Revert &quot;Enable mobile editor abandonment survey on enwiki&quot; (T424102)]], [[gerrit:1284702{{!}}Remove duplicate definition of EditCheckAction#isTagged (T425583)]], [[gerrit:1284703{{!}}Save action filtering info in ContentBranchNodeCheck#onDocumentChange (T425583)]] (duration: 06m 38s)
* 20:48 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 20:45 kemayo@deploy1003: esanders, kemayo: Continuing with deployment
* 20:44 kemayo@deploy1003: esanders, kemayo: Backport for [[gerrit:1284575{{!}}Revert &quot;Enable mobile editor abandonment survey on enwiki&quot; (T424102)]], [[gerrit:1284702{{!}}Remove duplicate definition of EditCheckAction#isTagged (T425583)]], [[gerrit:1284703{{!}}Save action filtering info in ContentBranchNodeCheck#onDocumentChange (T425583)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v
* 20:42 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1284575{{!}}Revert &quot;Enable mobile editor abandonment survey on enwiki&quot; (T424102)]], [[gerrit:1284702{{!}}Remove duplicate definition of EditCheckAction#isTagged (T425583)]], [[gerrit:1284703{{!}}Save action filtering info in ContentBranchNodeCheck#onDocumentChange (T425583)]]
* 20:41 Krinkle: krinkle@deploy1003$ mwscript deleteEqualMessages.php commonswiki
* 20:41 Krinkle: krinkle@deploy1003$ mwscript deleteEqualMessages.php nlwiki
* 20:34 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1021.eqiad.wmnet with reason: host reimage
* 20:30 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1021.eqiad.wmnet with reason: host reimage
* 20:29 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:28 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:14 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284687{{!}}Provide page context for LintErrorChecker (T419596)]], [[gerrit:1284771{{!}}Make email confirmation banner a standalone RL module (T425677)]] (duration: 07m 18s)
* 20:10 arlolra@deploy1003: arlolra, mmartorana: Continuing with deployment
* 20:10 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1021.eqiad.wmnet with OS trixie
* 20:09 arlolra@deploy1003: arlolra, mmartorana: Backport for [[gerrit:1284687{{!}}Provide page context for LintErrorChecker (T419596)]], [[gerrit:1284771{{!}}Make email confirmation banner a standalone RL module (T425677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:07 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1284687{{!}}Provide page context for LintErrorChecker (T419596)]], [[gerrit:1284771{{!}}Make email confirmation banner a standalone RL module (T425677)]]
* 20:02 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1022.eqiad.wmnet with OS trixie
* 19:59 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:57 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:09 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1022.eqiad.wmnet with OS trixie
* 19:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1022.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1022.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:51 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1022
* 18:49 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1022
* 18:49 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:49 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1022~] - vriley@cumin1003&quot;
* 18:49 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1022~] - vriley@cumin1003&quot;
* 18:45 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 18:26 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 18:26 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 18:25 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 18:24 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 18:22 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 18:22 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 18:21 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 18:21 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 18:20 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 18:19 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 18:19 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 18:18 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 18:17 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 18:06 cdanis@dns1005: END - running authdns-update
* 18:04 cdanis@dns1005: START - running authdns-update
* 18:02 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1237662{{!}}Profiler: Set explicit &quot;excimer-wall&quot; redis channel instead of concat]] (duration: 29m 24s)
* 18:02 brennen: 1.47.0-wmf.1 train status ([[phab:T423910|T423910]]): blockers resolved, rolling to all wikis
* 17:59 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 17:58 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 17:51 krinkle@deploy1003: krinkle: Continuing with deployment
* 17:50 krinkle@deploy1003: krinkle: Backport for [[gerrit:1237662{{!}}Profiler: Set explicit &quot;excimer-wall&quot; redis channel instead of concat]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:45 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 17:45 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 17:33 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1237662{{!}}Profiler: Set explicit &quot;excimer-wall&quot; redis channel instead of concat]]
* 17:32 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 17:32 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 17:06 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet,db1216.eqiad.wmnet with reason: restart
* 16:50 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2199.codfw.wmnet,db1245.eqiad.wmnet with reason: restart
* 16:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 16:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 16:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 16:35 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 16:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 16:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 16:32 jynus: restarting backup1-* database primary hosts
* 16:30 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2183.codfw.wmnet,db1204.eqiad.wmnet with reason: restart
* 16:25 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 36 hosts with reason: restart
* 16:14 sukhe@dns1004: END - running authdns-update
* 16:13 sukhe@dns1004: START - running authdns-update
* 16:13 sukhe@dns1004: START - running authdns-update
* 16:12 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 16:02 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and A:ulsfo and (A:dnsbox)
* 16:01 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 15:50 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on ms-backup[2003-2004].codfw.wmnet,ms-backup[1003-1004].eqiad.wmnet with reason: restart
* 15:44 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and A:ulsfo and (A:dnsbox)
* 15:32 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 15:32 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 15:31 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:31 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 15:31 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:31 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 15:24 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:24 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 39 hosts
* 15:22 sukhe@cumin1003: START - Cookbook sre.hosts.remove-downtime for 39 hosts
* 15:18 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:18 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:15 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4050.ulsfo.wmnet
* 15:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:12 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:06 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 15:06 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 15:06 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 15:05 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 15:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:03 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 15:03 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 15:03 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:01 akhatun: Deployed refinery using scap, then deployed onto hdfs
* 14:58 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
* 14:54 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 14:54 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 14:54 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 14:54 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 14:53 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 14:53 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 14:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:52 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 14:52 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 14:50 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:44 akhatun@deploy1003: Finished deploy [analytics/refinery@4734c67] (thin): Regular analytics weekly train THIN [analytics/refinery@4734c67c] (duration: 02m 01s)
* 14:43 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 14:43 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 14:42 akhatun@deploy1003: Started deploy [analytics/refinery@4734c67] (thin): Regular analytics weekly train THIN [analytics/refinery@4734c67c]
* 14:40 akhatun@deploy1003: Finished deploy [analytics/refinery@4734c67]: Regular analytics weekly train [analytics/refinery@4734c67c] (duration: 04m 38s)
* 14:40 jasmine@cumin2002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
* 14:37 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 14:36 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 14:36 akhatun@deploy1003: Started deploy [analytics/refinery@4734c67]: Regular analytics weekly train [analytics/refinery@4734c67c]
* 14:35 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 14:35 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 14:33 akhatun@deploy1003: Finished deploy [analytics/refinery@4734c67] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4734c67c] (duration: 01m 54s)
* 14:32 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: cluster=dnsbox,dc=ulsfo [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 14:32 slyngshede@dns1004: END - running authdns-update
* 14:32 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:31 akhatun@deploy1003: Started deploy [analytics/refinery@4734c67] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4734c67c]
* 14:31 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:31 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 14:30 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 14:30 slyngshede@dns1004: START - running authdns-update
* 14:30 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 14:30 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 14:30 akhatun: Deploying Refinery at {{Gerrit|4734c67}} for weekly deployment train
* 14:30 jmm@dns1004: END - running authdns-update
* 14:29 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 14:28 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 14:28 jmm@dns1004: START - running authdns-update
* 14:28 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:28 slyngshede@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating DNS snippets - slyngshede@cumin1003&quot;
* 14:28 slyngshede@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating DNS snippets - slyngshede@cumin1003&quot;
* 14:26 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 14:26 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 14:25 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 14:25 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 14:24 slyngshede@cumin1003: START - Cookbook sre.dns.netbox
* 14:12 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
* 14:12 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 14:12 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 14:10 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 14:10 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 13:53 jasmine@cumin2002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
* 13:34 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284553{{!}}Enable staggered rollout for IRS on enwiki (T424008)]], [[gerrit:1284569{{!}}Fix when user is considered exposed to the feature in the experiment (T424075)]] (duration: 09m 05s)
* 13:30 stran@deploy1003: stran: Continuing with deployment
* 13:27 stran@deploy1003: stran: Backport for [[gerrit:1284553{{!}}Enable staggered rollout for IRS on enwiki (T424008)]], [[gerrit:1284569{{!}}Fix when user is considered exposed to the feature in the experiment (T424075)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1284553{{!}}Enable staggered rollout for IRS on enwiki (T424008)]], [[gerrit:1284569{{!}}Fix when user is considered exposed to the feature in the experiment (T424075)]]
* 13:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 13:10 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284547{{!}}Remove the progress bar]], [[gerrit:1275467{{!}}mc: Set server, instead of host and port, for wgWikiLambdaObjectCaches (T423311)]] (duration: 06m 55s)
* 13:06 jforrester@deploy1003: rzl, jforrester, hartman: Continuing with deployment
* 13:05 jforrester@deploy1003: rzl, jforrester, hartman: Backport for [[gerrit:1284547{{!}}Remove the progress bar]], [[gerrit:1275467{{!}}mc: Set server, instead of host and port, for wgWikiLambdaObjectCaches (T423311)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:03 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1284547{{!}}Remove the progress bar]], [[gerrit:1275467{{!}}mc: Set server, instead of host and port, for wgWikiLambdaObjectCaches (T423311)]]
* 13:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=dns4004.wikimedia.org [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 12:58 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:55 sukhe@cumin1003: START - Cookbook sre.dns.netbox
* 12:51 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 12:51 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 12:51 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 12:50 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 12:45 sukhe@dns1004: FAIL - running authdns-update
* 12:44 sukhe@dns1004: START - running authdns-update
* 12:30 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1205.eqiad.wmnet with OS trixie
* 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host install5004.wikimedia.org
* 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host install5004.wikimedia.org with OS bookworm
* 12:23 slyngshede@dns1004: FAIL - running authdns-update
* 12:21 slyngshede@dns1004: START - running authdns-update
* 12:18 moritzm: installing init-system-helpers bugfix updates from Bookworm point release
* 12:18 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;add lswtest back as being planned won't work - cmooney@cumin1003&quot;
* 12:17 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;add lswtest back as being planned won't work - cmooney@cumin1003&quot;
* 12:12 slyngshede@dns1004: FAIL - running authdns-update
* 12:11 slyngshede@dns1004: START - running authdns-update
* 12:11 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 12:11 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 12:11 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: cluster=dnsbox,dc=ulsfo,service=authdns-update [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 12:08 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1205.eqiad.wmnet with reason: host reimage
* 12:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: after reimage to trixie
* 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on install5004.wikimedia.org with reason: host reimage
* 12:02 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 12:02 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 12:02 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1205.eqiad.wmnet with reason: host reimage
* 12:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on install5004.wikimedia.org with reason: host reimage
* 11:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1227: after reimage to trixie
* 11:47 root@cumin1003: START - Cookbook sre.hosts.reimage for host db1205.eqiad.wmnet with OS trixie
* 11:46 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1205.eqiad.wmnet with reason: reimage
* 11:43 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 11:43 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 11:40 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2184.codfw.wmnet with OS trixie
* 11:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
* 11:36 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 11:35 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 11:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
* 11:20 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2168: after reimage to trixie
* 11:19 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2184.codfw.wmnet with reason: host reimage
* 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2168.codfw.wmnet with OS trixie
* 11:16 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 11:15 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 11:15 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2184.codfw.wmnet with reason: host reimage
* 11:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92412 and previous config saved to /var/cache/conftool/dbconfig/20260507-111424-fceratto.json
* 11:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1227: after reimage to trixie
* 11:11 moritzm: instaling modsecurity-apache security updates
* 11:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1227.eqiad.wmnet with OS trixie
* 11:07 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host install5004.wikimedia.org with OS bookworm
* 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92409 and previous config saved to /var/cache/conftool/dbconfig/20260507-110415-fceratto.json
* 11:04 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 11:04 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 11:03 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 11:03 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:59 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 10:59 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:59 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 10:58 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db2184
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2184
* 10:57 root@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db2184
* 10:57 root@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2184.codfw.wmnet 129.32.192.10.in-addr.arpa 9.2.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:57 root@cumin1003: START - Cookbook sre.dns.wipe-cache db2184.codfw.wmnet 129.32.192.10.in-addr.arpa 9.2.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:57 root@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:57 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 10:57 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 10:57 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 10:57 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:57 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:56 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284592{{!}}Close Russian Wikinews (T421796)]] (duration: 08m 40s)
* 10:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2168.codfw.wmnet with reason: host reimage
* 10:54 root@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92407 and previous config saved to /var/cache/conftool/dbconfig/20260507-105407-fceratto.json
* 10:51 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 10:51 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2168.codfw.wmnet with reason: host reimage
* 10:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1284592{{!}}Close Russian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:49 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:49 root@cumin1003: START - Cookbook sre.hosts.move-vlan for host db2184
* 10:48 root@cumin1003: START - Cookbook sre.hosts.reimage for host db2184.codfw.wmnet with OS trixie
* 10:48 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1227.eqiad.wmnet with reason: host reimage
* 10:48 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 10:47 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 10:47 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1284592{{!}}Close Russian Wikinews (T421796)]]
* 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92406 and previous config saved to /var/cache/conftool/dbconfig/20260507-104359-fceratto.json
* 10:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1227.eqiad.wmnet with reason: host reimage
* 10:40 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2184.codfw.wmnet with reason: reimage
* 10:40 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 10:40 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 10:39 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 10:39 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:39 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:35 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:35 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92405 and previous config saved to /var/cache/conftool/dbconfig/20260507-103349-fceratto.json
* 10:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance
* 10:32 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2168.codfw.wmnet with OS trixie
* 10:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts hcaptcha-proxy5002.wikimedia.org
* 10:32 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:31 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: Reimage to Trixie
* 10:30 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2168: Reimage to Trixie
* 10:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2168.codfw.wmnet with reason: Reimage to Trixie
* 10:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2182: after reimage to trixie
* 10:28 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1227.eqiad.wmnet with OS trixie
* 10:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1227: Reimage to Trixie
* 10:26 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1227: Reimage to Trixie
* 10:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1227.eqiad.wmnet with reason: Reimage to Trixie
* 10:26 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1202: after reimage to trixie
* 10:21 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:21 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 10:20 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 10:16 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts hcaptcha-proxy5002.wikimedia.org
* 10:14 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 10:14 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 10:13 moritzm: rebalance ganti cluster in ulsfo following host reimages [[phab:T424686|T424686]]
* 10:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts hcaptcha-proxy5001.wikimedia.org
* 10:11 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:11 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 10:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast4006.wikimedia.org with OS trixie
* 10:10 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 10:04 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 10:03 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:03 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 09:59 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts hcaptcha-proxy5001.wikimedia.org
* 09:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast4006.wikimedia.org with reason: host reimage
* 09:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast4006.wikimedia.org with reason: host reimage
* 09:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2182: after reimage to trixie
* 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of hcaptcha-proxy4003.wikimedia.org to drbd
* 09:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2182.codfw.wmnet with OS trixie
* 09:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1202: after reimage to trixie
* 09:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1202.eqiad.wmnet with OS trixie
* 09:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
* 09:32 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of hcaptcha-proxy4003.wikimedia.org to drbd
* 09:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus4003.ulsfo.wmnet to drbd
* 09:25 elukey@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts sretest1006.eqiad.wmnet
* 09:24 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1006.eqiad.wmnet
* 09:19 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast4006.wikimedia.org with OS trixie
* 09:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
* 09:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM bast4006.wikimedia.org
* 09:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1202.eqiad.wmnet with reason: host reimage
* 09:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
* 09:11 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM bast4006.wikimedia.org
* 09:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2208: After reimage
* 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1202.eqiad.wmnet with reason: host reimage
* 08:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2182.codfw.wmnet with OS trixie
* 08:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1202.eqiad.wmnet with OS trixie
* 08:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1202: Reimage to Trixie
* 08:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2182: Reimage to Trixie
* 08:51 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2182: Reimage to Trixie
* 08:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2182.codfw.wmnet with reason: Reimage to Trixie
* 08:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1202: Reimage to Trixie
* 08:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1202.eqiad.wmnet with reason: Reimage to Trixie
* 08:42 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2144.codfw.wmnet
* 08:42 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:42 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2144.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 08:42 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2144.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 08:37 marostegui@cumin1003: START - Cookbook sre.dns.netbox
* 08:32 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2144.codfw.wmnet
* 08:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus4003.ulsfo.wmnet to drbd
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir4004.ulsfo.wmnet to drbd
* 08:28 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db2144 [[phab:T425522|T425522]]', diff saved to https://phabricator.wikimedia.org/P92389 and previous config saved to /var/cache/conftool/dbconfig/20260507-082822-marostegui.json
* 08:23 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2208: After reimage
* 08:23 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2208: After reimage
* 08:23 XioNoX: drmrs remove old v6 gateway IP
* 08:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: drmrs v6 gateway IPs change - ayounsi@cumin1003&quot;
* 08:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2208: After reimage
* 08:21 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: drmrs v6 gateway IPs change - ayounsi@cumin1003&quot;
* 08:17 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir4004.ulsfo.wmnet to drbd
* 08:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 08:12 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync
* 08:12 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync
* 08:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 08:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
* 08:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
* 08:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 08:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 07:54 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269465{{!}}search: add alt. completion indices to test keyword tokenizer (2/2) (T420427)]] (duration: 09m 46s)
* 07:49 dcausse@deploy1003: dcausse: Continuing with deployment
* 07:46 dcausse@deploy1003: dcausse: Backport for [[gerrit:1269465{{!}}search: add alt. completion indices to test keyword tokenizer (2/2) (T420427)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:44 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow4003.ulsfo.wmnet to drbd
* 07:44 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1269465{{!}}search: add alt. completion indices to test keyword tokenizer (2/2) (T420427)]]
* 07:32 moritzm: installing apache2 security updates
* 07:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow4003.ulsfo.wmnet to drbd
* 07:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM testvm2005.codfw.wmnet
* 07:23 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM testvm2005.codfw.wmnet
* 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:46 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:42 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4006.ulsfo.wmnet to cluster ulsfo02 and group 01
* 06:41 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti4006.ulsfo.wmnet to cluster ulsfo02 and group 01
* 06:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: after reimage to trixie
* 05:54 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2207: after reimage to trixie
* 05:51 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2207.codfw.wmnet with OS trixie
* 05:33 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2208.codfw.wmnet with OS trixie
* 05:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2207.codfw.wmnet with reason: host reimage
* 05:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2207.codfw.wmnet with reason: host reimage
* 05:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2208.codfw.wmnet with reason: host reimage
* 05:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2208.codfw.wmnet with reason: host reimage
* 05:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2207.codfw.wmnet with OS trixie
* 05:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: Reimage to Trixie
* 05:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2207: Reimage to Trixie
* 05:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2207.codfw.wmnet with reason: Reimage to Trixie
* 04:52 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2207 [[phab:T424848|T424848]]', diff saved to https://phabricator.wikimedia.org/P92383 and previous config saved to /var/cache/conftool/dbconfig/20260507-045219-marostegui.json
* 04:51 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2204 to s2 primary [[phab:T424848|T424848]]', diff saved to https://phabricator.wikimedia.org/P92382 and previous config saved to /var/cache/conftool/dbconfig/20260507-045141-marostegui.json
* 04:51 marostegui: Starting s2 codfw failover from db2207 to db2204 - [[phab:T424848|T424848]]
* 04:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T424848|T424848]]
* 04:46 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2204 with weight 0 [[phab:T424848|T424848]]', diff saved to https://phabricator.wikimedia.org/P92381 and previous config saved to /var/cache/conftool/dbconfig/20260507-044651-marostegui.json
* 04:46 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2208.codfw.wmnet with OS trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s)
* 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:15 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281485{{!}}Drop some unneeded wikinews configs (T421796)]] (duration: 12m 57s)
* 01:09 zabe@deploy1003: zabe: Continuing with deployment
* 01:09 zabe@deploy1003: zabe: Backport for [[gerrit:1281485{{!}}Drop some unneeded wikinews configs (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 01:02 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281485{{!}}Drop some unneeded wikinews configs (T421796)]]
* 01:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1021.eqiad.wmnet with OS trixie
* 00:43 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277783{{!}}Undeploy GoogleNewsSitemap (T421798)]] (duration: 33m 54s)
* 00:31 zabe@deploy1003: zabe: Continuing with deployment
* 00:29 zabe@deploy1003: zabe: Backport for [[gerrit:1277783{{!}}Undeploy GoogleNewsSitemap (T421798)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:10 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1277783{{!}}Undeploy GoogleNewsSitemap (T421798)]]

== 2026-05-06 ==
* 23:41 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1021.eqiad.wmnet with OS trixie
* 23:38 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1021.eqiad.wmnet with OS trixie
* 23:14 ladsgroup@deploy1003: Synchronized portals: Sync portals for removal of Wikinews (duration: 02m 22s)
* 23:12 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Sync portals for removal of Wikinews (duration: 06m 12s)
* 22:50 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284004{{!}}Close Spanish Wikinews (T421796)]] (duration: 07m 08s)
* 22:46 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:45 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1284004{{!}}Close Spanish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:43 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1284004{{!}}Close Spanish Wikinews (T421796)]]
* 22:33 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283872{{!}}Close English Wikinews (T421796)]] (duration: 06m 40s)
* 22:28 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:28 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283872{{!}}Close English Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283872{{!}}Close English Wikinews (T421796)]]
* 22:18 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1021.eqiad.wmnet with OS trixie
* 22:14 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:14 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283972{{!}}UBN fix: guard entry.serverTiming before forEach (T425591)]] (duration: 06m 25s)
* 22:11 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:11 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:10 cjming@deploy1003: cjming: Continuing with deployment
* 22:10 cjming@deploy1003: cjming: Backport for [[gerrit:1283972{{!}}UBN fix: guard entry.serverTiming before forEach (T425591)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:09 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:08 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:08 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1283972{{!}}UBN fix: guard entry.serverTiming before forEach (T425591)]]
* 22:06 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 22:05 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1021
* 22:04 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1021
* 21:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283953{{!}}Disable GNSM on dewikinews (T421798)]] (duration: 06m 56s)
* 21:48 zabe@deploy1003: zabe: Continuing with deployment
* 21:47 zabe@deploy1003: zabe: Backport for [[gerrit:1283953{{!}}Disable GNSM on dewikinews (T421798)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:45 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1283953{{!}}Disable GNSM on dewikinews (T421798)]]
* 21:31 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:29 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:28 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:27 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1012.eqiad.wmnet with OS trixie
* 21:26 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:24 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:22 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:17 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:15 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:14 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:12 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:11 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:11 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1021] - vriley@cumin1003&quot;
* 21:11 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1021] - vriley@cumin1003&quot;
* 21:07 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 21:06 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1021
* 21:05 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1021
* 21:04 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 20:29 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281526{{!}}Replace use of $wgRequest (T336703)]] (duration: 09m 12s)
* 20:25 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 20:24 catrope@deploy1003: catrope, somerandomdeveloper: Continuing with deployment
* 20:21 catrope@deploy1003: catrope, somerandomdeveloper: Backport for [[gerrit:1281526{{!}}Replace use of $wgRequest (T336703)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:19 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1281526{{!}}Replace use of $wgRequest (T336703)]]
* 20:14 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie
* 20:00 bking@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1012.eqiad.wmnet with OS trixie
* 19:57 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie
* 19:30 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS trixie
* 19:24 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1012.eqiad.wmnet with OS trixie
* 19:23 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4050.ulsfo.wmnet with OS trixie
* 19:14 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS bookworm
* 19:05 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 19:01 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 18:59 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 18:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 18:55 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 18:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 18:55 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 18:54 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 18:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 18:54 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 18:53 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 18:53 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:49 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie
* 18:48 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 18:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 18:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:42 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 18:42 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 18:42 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 18:41 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 18:40 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 18:40 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 18:39 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:37 dzahn@dns1005: END - running authdns-update
* 18:35 dzahn@dns1005: START - running authdns-update
* 18:33 brennen: 1.47.0-wmf.1 train status ([[phab:T423910|T423910]]): blockers resolved, rolling to group1
* 18:31 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp4050.ulsfo.wmnet with OS trixie
* 18:29 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS bookworm
* 18:02 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-23-ulsfo
* 18:01 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-23-ulsfo
* 17:59 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: restart to test fixes from [[phab:T425301|T425301]] - bking@cumin2002
* 17:55 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device asw1-23-ulsfo
* 17:55 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-23-ulsfo
* 17:37 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 17:32 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 17:31 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 17:28 topranks: rebooting asw1-23-ulsfo to upgrade SR-Linux OS on switch [[phab:T408892|T408892]]
* 17:27 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw1-23-ulsfo,asw1-23-ulsfo IPv6 with reason: upgrading sr-linux on asw1-23-ulsfo
* 17:20 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 17:18 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
* 17:18 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
* 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
* 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
* 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
* 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
* 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
* 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
* 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 17:06 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 17:02 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on 39 hosts with reason: ulsfo depooled for switch work
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw1-22-ulsfo,asw1-22-ulsfo IPv6 with reason: upgrading sr-linux on asw1-23-ulsfo
* 16:52 topranks: rebooting asw1-22-ulsfo to upgrade SR-Linux OS on switch [[phab:T408892|T408892]]
* 16:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:40 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS trixie
* 16:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:37 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS bookworm
* 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:28 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS bookworm
* 16:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:28 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:09 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 16:04 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4004.wikimedia.org with reason: host reimage
* 15:58 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 15:57 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns4004.wikimedia.org with reason: host reimage
* 15:38 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS bookworm
* 15:35 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS bookworm
* 15:30 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=0) Change Confluent distribution for Kafka A:kafka-main-eqiad cluster: Change Confluent distribution.
* 15:08 sukhe: sudo cumin -b1 -s5 &quot;C:bird and not dns4004*&quot; &quot;run-puppet-agent --enable 'merging CR 1282958'&quot;
* 15:08 jasmine@cumin2002: START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-main-eqiad cluster: Change Confluent distribution.
* 15:06 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283805{{!}}Close Chinese Wikinews (T421796)]] (duration: 06m 41s)
* 15:02 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 15:01 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:01 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283805{{!}}Close Chinese Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:59 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283805{{!}}Close Chinese Wikinews (T421796)]]
* 14:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts durum5002.eqsin.wmnet
* 14:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:53 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS trixie
* 14:51 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:45 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 14:36 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:36 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:35 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.wikimedia.org [reason: testing bird change]
* 14:31 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.wikimedia.org [reason: testing bird change]
* 14:30 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283050{{!}}Add user_groups to editAttemptStep schema (T424010)]] (duration: 11m 16s)
* 14:28 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
* 14:26 kharlan@deploy1003: kharlan: Continuing with deployment
* 14:25 sukhe: sudo cumin &quot;C:bird&quot; &quot;disable-puppet 'merging CR 1282958'&quot;
* 14:23 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
* 14:22 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:21 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:21 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1283050{{!}}Add user_groups to editAttemptStep schema (T424010)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:21 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts durum5002.eqsin.wmnet
* 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4008.ulsfo.wmnet with OS bookworm
* 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 14:20 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 14:20 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:19 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1283050{{!}}Add user_groups to editAttemptStep schema (T424010)]]
* 14:19 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:18 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts durum5001.eqsin.wmnet
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:15 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283783{{!}}Close German Wikinews (T421796)]] (duration: 06m 40s)
* 14:13 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:13 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:12 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:12 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:11 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4046.ulsfo.wmnet with OS trixie
* 14:10 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 14:10 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:10 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283783{{!}}Close German Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:10 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:09 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:08 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:08 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283783{{!}}Close German Wikinews (T421796)]]
* 14:08 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 14:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283751{{!}}Close French Wikinews (T421796)]] (duration: 11m 28s)
* 14:02 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts durum5001.eqsin.wmnet
* 14:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage
* 13:56 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 13:56 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage
* 13:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283751{{!}}Close French Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:55 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS trixie
* 13:53 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: restart to test fixes from [[phab:T425301|T425301]] - bking@cumin2002
* 13:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1194: after reimage to trixie
* 13:51 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283751{{!}}Close French Wikinews (T421796)]]
* 13:45 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4046.ulsfo.wmnet with reason: host reimage
* 13:45 jgreen@dns1004: END - running authdns-update
* 13:44 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283028{{!}}Add messages related to mandatory 2FA for more groups (T423119)]] (duration: 30m 53s)
* 13:44 jgreen@dns1004: START - running authdns-update
* 13:41 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4046.ulsfo.wmnet with reason: host reimage
* 13:39 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 13:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti4008.ulsfo.wmnet with OS bookworm
* 13:35 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) ganeti4008.ulsfo.wmnet on all recursors
* 13:34 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti4008.ulsfo.wmnet on all recursors
* 13:32 alexsanford@deploy1003: alexsanford: Continuing with deployment
* 13:32 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:31 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 13:31 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1283028{{!}}Add messages related to mandatory 2FA for more groups (T423119)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:28 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:28 jmm@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:26 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 13:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:20 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:19 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) ganeti4008.mgmt.ulsfo.wmnet on all recursors
* 13:19 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS trixie
* 13:19 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti4008.mgmt.ulsfo.wmnet on all recursors
* 13:19 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:18 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for ganeti4008 mgmt - cmooney@cumin1003&quot;
* 13:18 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for ganeti4008 mgmt - cmooney@cumin1003&quot;
* 13:15 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 13:14 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 13:14 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 13:13 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1283028{{!}}Add messages related to mandatory 2FA for more groups (T423119)]]
* 13:12 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS trixie
* 13:11 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:08 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 13:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1194: after reimage to trixie
* 13:05 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 13:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1194.eqiad.wmnet with OS trixie
* 12:49 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS trixie
* 12:45 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 12:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2012.codfw.wmnet with OS trixie
* 12:39 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 12:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
* 12:36 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: update
* 12:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
* 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2012.codfw.wmnet with reason: host reimage
* 12:21 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2012.codfw.wmnet with reason: host reimage
* 12:20 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1194.eqiad.wmnet with OS trixie
* 12:20 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4048.ulsfo.wmnet with OS trixie
* 12:16 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4050.ulsfo.wmnet with OS trixie
* 12:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 12:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 12:14 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2011.codfw.wmnet with OS trixie
* 12:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283735{{!}}Close Polish Wikinews (T421796)]] (duration: 06m 28s)
* 12:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 12:07 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 12:07 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283735{{!}}Close Polish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:05 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb2012.codfw.wmnet with OS trixie
* 12:05 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283735{{!}}Close Polish Wikinews (T421796)]]
* 12:05 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 11:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2011.codfw.wmnet with reason: host reimage
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet
* 11:53 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4048.ulsfo.wmnet with reason: host reimage
* 11:50 moritzm: installing openjdk-17 security updates
* 11:50 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 11:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92374 and previous config saved to /var/cache/conftool/dbconfig/20260506-114919-fceratto.json
* 11:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet
* 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1194: Reimage to Trixie
* 11:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2160.codfw.wmnet with reason: Reboot
* 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1194: Reimage to Trixie
* 11:44 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2011.codfw.wmnet with reason: host reimage
* 11:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1194.eqiad.wmnet with reason: Reimage to Trixie
* 11:42 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4048.ulsfo.wmnet with reason: host reimage
* 11:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4006.ulsfo.wmnet with OS bookworm
* 11:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 11:41 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92372 and previous config saved to /var/cache/conftool/dbconfig/20260506-113910-fceratto.json
* 11:30 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb2011.codfw.wmnet with OS trixie
* 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92371 and previous config saved to /var/cache/conftool/dbconfig/20260506-112903-fceratto.json
* 11:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 11:20 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4050.ulsfo.wmnet with OS trixie
* 11:19 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4048.ulsfo.wmnet with OS trixie
* 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92370 and previous config saved to /var/cache/conftool/dbconfig/20260506-111854-fceratto.json
* 11:14 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS trixie
* 11:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4042.ulsfo.wmnet with OS trixie
* 11:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Reboot
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4006.ulsfo.wmnet with reason: host reimage
* 10:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4006.ulsfo.wmnet with reason: host reimage
* 10:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage
* 10:44 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4042.ulsfo.wmnet with reason: host reimage
* 10:40 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage
* 10:39 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4042.ulsfo.wmnet with reason: host reimage
* 10:33 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti4006.ulsfo.wmnet with OS bookworm
* 10:29 jmm@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:23 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:22 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92369 and previous config saved to /var/cache/conftool/dbconfig/20260506-101836-fceratto.json
* 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92368 and previous config saved to /var/cache/conftool/dbconfig/20260506-101808-fceratto.json
* 10:17 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS trixie
* 10:16 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4042.ulsfo.wmnet with OS trixie
* 10:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4040.ulsfo.wmnet with OS trixie
* 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92367 and previous config saved to /var/cache/conftool/dbconfig/20260506-100800-fceratto.json
* 09:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92366 and previous config saved to /var/cache/conftool/dbconfig/20260506-095752-fceratto.json
* 09:55 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92365 and previous config saved to /var/cache/conftool/dbconfig/20260506-094744-fceratto.json
* 09:45 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4040.ulsfo.wmnet with reason: host reimage
* 09:40 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4040.ulsfo.wmnet with reason: host reimage
* 09:38 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:32 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:31 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:29 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:29 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006/8 mgmt - ayounsi@cumin1003&quot;
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92364 and previous config saved to /var/cache/conftool/dbconfig/20260506-092414-fceratto.json
* 09:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance
* 09:23 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006/8 mgmt - ayounsi@cumin1003&quot;
* 09:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92363 and previous config saved to /var/cache/conftool/dbconfig/20260506-092345-fceratto.json
* 09:17 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 09:17 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4040.ulsfo.wmnet with OS trixie
* 09:16 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
* 09:15 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: update
* 09:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repool ms2 [[phab:T418979|T418979]]ç', diff saved to https://phabricator.wikimedia.org/P92362 and previous config saved to /var/cache/conftool/dbconfig/20260506-091513-marostegui.json
* 09:14 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 09:14 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2253: Replacing HW [[phab:T418979|T418979]]
* 09:14 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99)
* 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
* 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2253: Replacing HW [[phab:T418979|T418979]]
* 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039', diff saved to https://phabricator.wikimedia.org/P92361 and previous config saved to /var/cache/conftool/dbconfig/20260506-091337-fceratto.json
* 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039', diff saved to https://phabricator.wikimedia.org/P92360 and previous config saved to /var/cache/conftool/dbconfig/20260506-090329-fceratto.json
* 09:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281894{{!}}Correctly support new file tables in RevisionDeleteUser (T424553)]] (duration: 08m 44s)
* 08:59 zabe@deploy1003: zabe: Continuing with deployment
* 08:56 zabe@deploy1003: zabe: Backport for [[gerrit:1281894{{!}}Correctly support new file tables in RevisionDeleteUser (T424553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281894{{!}}Correctly support new file tables in RevisionDeleteUser (T424553)]]
* 08:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92359 and previous config saved to /var/cache/conftool/dbconfig/20260506-085321-fceratto.json
* 08:43 marostegui@cumin1003: dbctl commit (dc=all): 'Add db2253 to ms2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92358 and previous config saved to /var/cache/conftool/dbconfig/20260506-084337-marostegui.json
* 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92357 and previous config saved to /var/cache/conftool/dbconfig/20260506-083841-fceratto.json
* 08:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance
* 08:29 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2208.codfw.wmnet with OS trixie
* 08:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2208.codfw.wmnet with OS trixie
* 08:08 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2208.codfw.wmnet with OS trixie
* 08:06 awight: EU morning deployment is done
* 08:00 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2144.codfw.wmnet,db1151.eqiad.wmnet with reason: Replacing hw
* 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2144: Replacing HW [[phab:T418979|T418979]]
* 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 07:58 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
* 07:58 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2144: Replacing HW [[phab:T418979|T418979]]
* 07:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2208.codfw.wmnet with OS trixie
* 07:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283101{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]], [[gerrit:1283037{{!}}search: fix alt. completion indices to test keyword tokenizer (T420427)]], [[gerrit:1283041{{!}}search: enable Latin-to-Devanagari transliteration second-chance (T425018)]] (duration: 08m 58s)
* 07:36 awight@deploy1003: wmde-fisch, awight, dcausse: Continuing with deployment
* 07:33 awight@deploy1003: wmde-fisch, awight, dcausse: Backport for [[gerrit:1283101{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]], [[gerrit:1283037{{!}}search: fix alt. completion indices to test keyword tokenizer (T420427)]], [[gerrit:1283041{{!}}search: enable Latin-to-Devanagari transliteration second-chance (T425018)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can
* 07:31 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1283101{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]], [[gerrit:1283037{{!}}search: fix alt. completion indices to test keyword tokenizer (T420427)]], [[gerrit:1283041{{!}}search: enable Latin-to-Devanagari transliteration second-chance (T425018)]]
* 07:26 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283033{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]] (duration: 07m 37s)
* 07:22 awight@deploy1003: awight, lilients: Continuing with deployment
* 07:21 awight@deploy1003: awight, lilients: Backport for [[gerrit:1283033{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:19 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1283033{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]]
* 07:14 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4008.ulsfo.wmnet
* 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4008.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4008.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 06:55 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 06:54 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1191: after reimage to trixie
* 06:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1189: after reimage to trixie
* 06:48 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti4008.ulsfo.wmnet
* 06:48 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4006.ulsfo.wmnet
* 06:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 06:30 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 06:26 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti4006.ulsfo.wmnet
* 05:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2208.codfw.wmnet with reason: Idrac issues [[phab:T425506|T425506]]
* 05:43 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
* 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
* 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
* 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
* 05:26 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2208: Reimage to Trixie
* 05:26 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2208: Reimage to Trixie
* 05:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2208.codfw.wmnet with reason: Reimage to Trixie
* 05:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2208: Reimage to Trixie
* 05:24 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2208: Reimage to Trixie
* 05:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2208.codfw.wmnet with reason: Reimage to Trixie
* 05:23 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1191.eqiad.wmnet with OS trixie
* 05:22 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1191: Reimage to Trixie
* 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1191: Reimage to Trixie
* 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Reimage to Trixie
* 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1189.eqiad.wmnet with OS trixie
* 05:14 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1189: Reimage to Trixie
* 05:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1189: Reimage to Trixie
* 05:14 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1189.eqiad.wmnet with reason: Reimage to Trixie
* 05:11 marostegui@dns1004: END - running authdns-update
* 05:09 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1189 [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92345 and previous config saved to /var/cache/conftool/dbconfig/20260506-050948-marostegui.json
* 05:09 marostegui@dns1004: START - running authdns-update
* 05:08 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1223 to s3 primary and set section read-write [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92344 and previous config saved to /var/cache/conftool/dbconfig/20260506-050816-marostegui.json
* 05:07 marostegui@cumin1003: dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92343 and previous config saved to /var/cache/conftool/dbconfig/20260506-050755-marostegui.json
* 05:06 marostegui: Starting s3 eqiad failover from db1189 to db1223 - [[phab:T425318|T425318]]
* 05:03 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 [[phab:T425318|T425318]]
* 05:03 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1223 with weight 0 [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92342 and previous config saved to /var/cache/conftool/dbconfig/20260506-050342-marostegui.json
* 03:28 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 03:27 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 03:27 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 03:26 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 37s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:05 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1001.eqiad.wmnet with OS trixie
* 00:54 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283061{{!}}Close Dutch Wikinews (T421796)]] (duration: 06m 26s)
* 00:49 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 00:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283061{{!}}Close Dutch Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283061{{!}}Close Dutch Wikinews (T421796)]]
* 00:45 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1001.eqiad.wmnet with reason: host reimage
* 00:41 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1001.eqiad.wmnet with reason: host reimage
* 00:27 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283117{{!}}Close Italian Wikinews (T421796)]] (duration: 07m 26s)
* 00:25 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1001
* 00:25 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1001
* 00:24 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1001.eqiad.wmnet with OS trixie
* 00:23 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 00:21 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283117{{!}}Close Italian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283117{{!}}Close Italian Wikinews (T421796)]]

== 2026-05-05 ==
* 23:31 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 23:30 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update ip addresses for nodes in rack 23 - pt1979@cumin2002&quot;
* 23:30 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update ip addresses for nodes in rack 23 - pt1979@cumin2002&quot;
* 23:26 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 22:54 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283104{{!}}Close Arabic Wikinews (T421796)]] (duration: 06m 58s)
* 22:50 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283104{{!}}Close Arabic Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283104{{!}}Close Arabic Wikinews (T421796)]]
* 22:43 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283103{{!}}Close Ukrainian Wikinews (T421796)]] (duration: 06m 28s)
* 22:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:39 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283103{{!}}Close Ukrainian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:37 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283103{{!}}Close Ukrainian Wikinews (T421796)]]
* 22:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283100{{!}}Close Romanian Wikinews (T421796)]] (duration: 07m 56s)
* 22:22 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283100{{!}}Close Romanian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283100{{!}}Close Romanian Wikinews (T421796)]]
* 22:16 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283099{{!}}Close Serbian Wikinews (T421796)]] (duration: 06m 45s)
* 22:12 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:11 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283099{{!}}Close Serbian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:09 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283099{{!}}Close Serbian Wikinews (T421796)]]
* 22:05 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283066{{!}}Close Persian Wikinews (T421796)]] (duration: 11m 07s)
* 21:59 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 21:58 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283066{{!}}Close Persian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:54 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283066{{!}}Close Persian Wikinews (T421796)]]
* 21:49 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281501{{!}}Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056{{!}}Legacy parser no longer varies by user thumbnail size. (T417513)]] (duration: 32m 55s)
* 21:36 arlolra@deploy1003: jdlrobson, mmartorana, arlolra: Continuing with deployment
* 21:33 arlolra@deploy1003: jdlrobson, mmartorana, arlolra: Backport for [[gerrit:1281501{{!}}Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056{{!}}Legacy parser no longer varies by user thumbnail size. (T417513)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:16 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1281501{{!}}Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056{{!}}Legacy parser no longer varies by user thumbnail size. (T417513)]]
* 20:59 dancy@deploy1003: Installation of scap version &quot;4.262.1&quot; completed for 2 hosts
* 20:57 dancy@deploy1003: Installing scap version &quot;4.262.1&quot; for 2 host(s)
* 20:57 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282930{{!}}hCaptcha: Add diagnostic context to script load error logs (T424496)]], [[gerrit:1282397{{!}}sectionCollapsing: Scroll to fragment target on init (T425290)]], [[gerrit:1282804{{!}}Errors added below ref list dirty when not responsive (T384599)]] (duration: 10m 59s)
* 20:52 arlolra@deploy1003: mpostoronca, h2o, awight, arlolra: Continuing with deployment
* 20:48 arlolra@deploy1003: mpostoronca, h2o, awight, arlolra: Backport for [[gerrit:1282930{{!}}hCaptcha: Add diagnostic context to script load error logs (T424496)]], [[gerrit:1282397{{!}}sectionCollapsing: Scroll to fragment target on init (T425290)]], [[gerrit:1282804{{!}}Errors added below ref list dirty when not responsive (T384599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve
* 20:46 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1282930{{!}}hCaptcha: Add diagnostic context to script load error logs (T424496)]], [[gerrit:1282397{{!}}sectionCollapsing: Scroll to fragment target on init (T425290)]], [[gerrit:1282804{{!}}Errors added below ref list dirty when not responsive (T384599)]]
* 20:41 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4038.ulsfo.wmnet with OS trixie
* 20:22 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283082{{!}}Enable WikiLove on shwiki (T424891)]], [[gerrit:1276814{{!}}Add wikibase.v1 module to the sandbox were it is present (T422403)]] (duration: 10m 30s)
* 20:20 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1002.eqiad.wmnet with OS trixie
* 20:18 arlolra@deploy1003: aaron, neriah, arlolra: Continuing with deployment
* 20:14 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4038.ulsfo.wmnet with reason: host reimage
* 20:13 arlolra@deploy1003: aaron, neriah, arlolra: Backport for [[gerrit:1283082{{!}}Enable WikiLove on shwiki (T424891)]], [[gerrit:1276814{{!}}Add wikibase.v1 module to the sandbox were it is present (T422403)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:12 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1283082{{!}}Enable WikiLove on shwiki (T424891)]], [[gerrit:1276814{{!}}Add wikibase.v1 module to the sandbox were it is present (T422403)]]
* 20:10 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:07 pt1979@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4038.ulsfo.wmnet with reason: host reimage
* 20:04 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: host reimage
* 19:57 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: host reimage
* 19:55 herron@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 19:55 herron@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 19:54 root@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
* 19:54 root@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
* 19:45 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 19:41 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1002
* 19:41 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging1002
* 19:41 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 19:41 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:39 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging1002
* 19:39 herron@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1002.eqiad.wmnet 142.32.64.10.in-addr.arpa 2.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 19:39 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging1002.eqiad.wmnet 142.32.64.10.in-addr.arpa 2.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 19:39 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:39 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1002 - herron@cumin1003&quot;
* 19:38 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1002 - herron@cumin1003&quot;
* 19:32 herron@cumin1003: START - Cookbook sre.dns.netbox
* 19:31 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:30 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:28 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1002
* 19:27 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1002.eqiad.wmnet with OS trixie
* 19:17 dancy@deploy1003: Installation of scap version &quot;4.262.0&quot; completed for 2 hosts
* 19:15 dancy@deploy1003: Installing scap version &quot;4.262.0&quot; for 2 host(s)
* 19:15 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: rebooting firewall in desperation
* 19:14 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 19:05 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;set correct vlan group in netbox for new ulsfo vlans - cmooney@cumin1003 - [[phab:T408892|T408892]]&quot;
* 19:05 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;set correct vlan group in netbox for new ulsfo vlans - cmooney@cumin1003 - [[phab:T408892|T408892]]&quot;
* 19:04 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie
* 19:03 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283063{{!}}Close Swedish Wikinews (T421796)]] (duration: 10m 59s)
* 18:56 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 18:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283063{{!}}Close Swedish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:52 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283063{{!}}Close Swedish Wikinews (T421796)]]
* 18:49 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 18:48 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:48 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:48 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:47 brennen@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.1  refs [[phab:T423910|T423910]] (duration: 36m 04s)
* 18:44 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie
* 18:44 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:44 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:44 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:44 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:40 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:30 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 18:25 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie
* 18:14 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device pfw1a-codfw
* 18:13 cmooney@cumin1003: START - Cookbook sre.network.tls for network device pfw1a-codfw
* 18:13 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 18:11 brennen@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 18:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device pfw1a-eqiad
* 18:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device pfw1a-eqiad
* 18:06 brennen: 1.47.0-wmf.1 train status ([[phab:T423910|T423910]]): no current blockers, rolling to group0
* 18:04 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1003.eqiad.wmnet with OS trixie
* 17:44 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: host reimage
* 17:38 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: host reimage
* 17:33 herron@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 17:32 herron@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 17:23 root@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
* 17:23 root@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
* 17:21 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1003
* 17:21 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging1003
* 17:21 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging1003
* 17:20 herron@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:19 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:16 herron@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:15 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:15 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:15 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1003 - herron@cumin1003&quot;
* 17:15 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1003 - herron@cumin1003&quot;
* 17:12 herron@cumin1003: START - Cookbook sre.dns.netbox
* 17:09 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1003
* 17:08 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1003.eqiad.wmnet with OS trixie
* 17:05 sukhe: sudo cumin -b11 &quot;A:cp and not P&lt;nowiki&gt;{&lt;/nowiki&gt;cp2041* or cp2042*&lt;nowiki&gt;}&lt;/nowiki&gt; and not A:ulsfo&quot; &quot;run-puppet-agent --enable 'merging CR 1282979'&quot;
* 16:58 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049{{!}}Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]] (duration: 07m 25s)
* 16:53 sbassett@deploy1003: mstyles, sbassett: Continuing with deployment
* 16:52 sbassett@deploy1003: mstyles, sbassett: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049{{!}}Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdeb
* 16:50 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049{{!}}Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]]
* 16:38 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]]
* 16:19 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: sync
* 16:19 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: sync
* 16:19 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync
* 16:18 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: sync
* 16:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283040{{!}}Close Japanese Wikinews (T421796)]] (duration: 06m 16s)
* 16:07 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 16:07 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283040{{!}}Close Japanese Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:05 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283040{{!}}Close Japanese Wikinews (T421796)]]
* 16:01 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283039{{!}}Close Korean Wikinews (T421796)]] (duration: 07m 53s)
* 15:57 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:55 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: sync
* 15:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283039{{!}}Close Korean Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:55 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: sync
* 15:55 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync
* 15:54 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: sync
* 15:53 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283039{{!}}Close Korean Wikinews (T421796)]]
* 15:52 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283038{{!}}Close Finnish Wikinews (T421796)]] (duration: 06m 12s)
* 15:48 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:47 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283038{{!}}Close Finnish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:46 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283038{{!}}Close Finnish Wikinews (T421796)]]
* 15:42 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync
* 15:42 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync
* 15:39 dzahn@dns1005: END - running authdns-update
* 15:38 mutante: deleting mwmaint.discovery.wmnet DNS entry - the hosts behind it dont exist anymore
* 15:37 dzahn@dns1005: START - running authdns-update
* 15:24 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 15:24 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 15:21 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283024{{!}}Close Czech Wikinews (T421796)]] (duration: 06m 17s)
* 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92340 and previous config saved to /var/cache/conftool/dbconfig/20260505-151930-fceratto.json
* 15:16 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:16 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283024{{!}}Close Czech Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283024{{!}}Close Czech Wikinews (T421796)]]
* 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P92339 and previous config saved to /var/cache/conftool/dbconfig/20260505-150921-fceratto.json
* 15:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283003{{!}}Close Tamil Wikinews (T421796)]] (duration: 07m 06s)
* 15:07 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:04 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:03 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283003{{!}}Close Tamil Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283003{{!}}Close Tamil Wikinews (T421796)]]
* 14:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283002{{!}}fix: wrong property name action_data (T425425)]] (duration: 07m 48s)
* 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P92338 and previous config saved to /var/cache/conftool/dbconfig/20260505-145913-fceratto.json
* 14:58 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 14:55 urbanecm@deploy1003: urbanecm: Continuing with deployment
* 14:53 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1283002{{!}}fix: wrong property name action_data (T425425)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92337 and previous config saved to /var/cache/conftool/dbconfig/20260505-145231-fceratto.json
* 14:51 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1283002{{!}}fix: wrong property name action_data (T425425)]]
* 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92336 and previous config saved to /var/cache/conftool/dbconfig/20260505-144905-fceratto.json
* 14:44 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1004.eqiad.wmnet with OS trixie
* 14:42 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P92335 and previous config saved to /var/cache/conftool/dbconfig/20260505-144223-fceratto.json
* 14:42 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 14:41 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:41 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92334 and previous config saved to /var/cache/conftool/dbconfig/20260505-144029-fceratto.json
* 14:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance
* 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92333 and previous config saved to /var/cache/conftool/dbconfig/20260505-143958-fceratto.json
* 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P92332 and previous config saved to /var/cache/conftool/dbconfig/20260505-143214-fceratto.json
* 14:30 jmm@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=config-master,name=eqiad
* 14:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P92331 and previous config saved to /var/cache/conftool/dbconfig/20260505-142949-fceratto.json
* 14:28 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1004.eqiad.wmnet with reason: host reimage
* 14:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM config-master1001.eqiad.wmnet
* 14:25 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1004.eqiad.wmnet with reason: host reimage
* 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM config-master1001.eqiad.wmnet
* 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92329 and previous config saved to /var/cache/conftool/dbconfig/20260505-142206-fceratto.json
* 14:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P92328 and previous config saved to /var/cache/conftool/dbconfig/20260505-141941-fceratto.json
* 14:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 14:11 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1004
* 14:10 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1004
* 14:10 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1004.eqiad.wmnet with OS trixie
* 14:09 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1015.eqiad.wmnet
* 14:09 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:09 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:09 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92327 and previous config saved to /var/cache/conftool/dbconfig/20260505-140933-fceratto.json
* 14:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 14:07 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 14:07 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 14:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 14:05 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 14:05 jmm@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=config-master,name=eqiad
* 14:05 jmm@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=config-master,name=codfw
* 14:04 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync
* 14:04 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync
* 14:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 14:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 14:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 14:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 14:03 Lucas_WMDE: UTC afternoon backport+config window done
* 14:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM config-master2001.codfw.wmnet
* 14:02 jasmine@cumin2002: END (FAIL) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=99) Change Confluent distribution for Kafka A:kafka-main-codfw cluster: Change Confluent distribution.
* 14:01 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1015.eqiad.wmnet
* 14:01 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1014.eqiad.wmnet
* 14:01 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:01 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:01 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92326 and previous config saved to /var/cache/conftool/dbconfig/20260505-140047-fceratto.json
* 14:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance
* 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92325 and previous config saved to /var/cache/conftool/dbconfig/20260505-140016-fceratto.json
* 13:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1227: Repooling
* 13:59 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:59 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM config-master2001.codfw.wmnet
* 13:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:55 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 13:54 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282988{{!}}Close Portuguese Wikinews (T421796)]] (duration: 06m 22s)
* 13:50 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1014.eqiad.wmnet
* 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P92323 and previous config saved to /var/cache/conftool/dbconfig/20260505-135008-fceratto.json
* 13:50 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 13:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282988{{!}}Close Portuguese Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:49 jmm@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=config-master,name=codfw
* 13:47 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1011.eqiad.wmnet
* 13:47 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:47 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1011.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282988{{!}}Close Portuguese Wikinews (T421796)]]
* 13:47 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1011.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92321 and previous config saved to /var/cache/conftool/dbconfig/20260505-134522-fceratto.json
* 13:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
* 13:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1227: Repooling
* 13:44 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 13:43 jasmine@cumin2002: START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-main-codfw cluster: Change Confluent distribution.
* 13:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92319 and previous config saved to /var/cache/conftool/dbconfig/20260505-134257-fceratto.json
* 13:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P92318 and previous config saved to /var/cache/conftool/dbconfig/20260505-134000-fceratto.json
* 13:37 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1011.eqiad.wmnet
* 13:37 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1010.eqiad.wmnet
* 13:37 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:37 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1010.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:37 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1010.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:33 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 13:30 Msz2001: UTC afternoon backport window done
* 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92317 and previous config saved to /var/cache/conftool/dbconfig/20260505-132952-fceratto.json
* 13:27 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1010.eqiad.wmnet
* 13:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:23 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:23 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270882{{!}}Remove temporary `wgOAuth2UsePrefixedSub` feature flag (T417690)]], [[gerrit:1271969{{!}}Move privileged global and local group handling to WikimediaCustomizations (T418507)]], [[gerrit:1281964{{!}}Add Akan (ak) to wmgExtraLanguageNames by default (T333765 T425256)]] (duration: 08m 37s)
* 13:23 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on dborch1002.wikimedia.org with reason: [[phab:T416582|T416582]]
* 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92316 and previous config saved to /var/cache/conftool/dbconfig/20260505-132002-fceratto.json
* 13:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance
* 13:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92315 and previous config saved to /var/cache/conftool/dbconfig/20260505-131931-fceratto.json
* 13:19 mszwarc@deploy1003: mszwarc, jhsoby, matmarex, d3r1ck01: Continuing with deployment
* 13:16 mszwarc@deploy1003: mszwarc, jhsoby, matmarex, d3r1ck01: Backport for [[gerrit:1270882{{!}}Remove temporary `wgOAuth2UsePrefixedSub` feature flag (T417690)]], [[gerrit:1271969{{!}}Move privileged global and local group handling to WikimediaCustomizations (T418507)]], [[gerrit:1281964{{!}}Add Akan (ak) to wmgExtraLanguageNames by default (T333765 T425256)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug
* 13:15 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1270882{{!}}Remove temporary `wgOAuth2UsePrefixedSub` feature flag (T417690)]], [[gerrit:1271969{{!}}Move privileged global and local group handling to WikimediaCustomizations (T418507)]], [[gerrit:1281964{{!}}Add Akan (ak) to wmgExtraLanguageNames by default (T333765 T425256)]]
* 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet
* 13:11 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282850{{!}}Switch 'autoconfirmed' to use APCOND_AGE_FROM_EDIT on certain wikis (T418484)]] (duration: 07m 55s)
* 13:11 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 13:11 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet
* 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P92314 and previous config saved to /var/cache/conftool/dbconfig/20260505-130923-fceratto.json
* 13:07 mszwarc@deploy1003: mszwarc: Continuing with deployment
* 13:05 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1282850{{!}}Switch 'autoconfirmed' to use APCOND_AGE_FROM_EDIT on certain wikis (T418484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1282850{{!}}Switch 'autoconfirmed' to use APCOND_AGE_FROM_EDIT on certain wikis (T418484)]]
* 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P92313 and previous config saved to /var/cache/conftool/dbconfig/20260505-125915-fceratto.json
* 12:56 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282970{{!}}Close Esperanto Wikinews (T421796)]] (duration: 07m 23s)
* 12:52 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 12:50 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282970{{!}}Close Esperanto Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282970{{!}}Close Esperanto Wikinews (T421796)]]
* 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92312 and previous config saved to /var/cache/conftool/dbconfig/20260505-124907-fceratto.json
* 12:44 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1280226{{!}}loggedOutWarning: instrument browser navigation and tab close (T421518)]] (duration: 03m 56s)
* 12:43 sgimeno@deploy1003: sgimeno: Continuing with deployment
* 12:42 moritzm: installing node-tar security updates
* 12:41 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1280226{{!}}loggedOutWarning: instrument browser navigation and tab close (T421518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:40 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1280226{{!}}loggedOutWarning: instrument browser navigation and tab close (T421518)]]
* 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2240 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92311 and previous config saved to /var/cache/conftool/dbconfig/20260505-124041-fceratto.json
* 12:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance
* 12:36 moritzm: installing imagemagick security updates
* 12:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2239.codfw.wmnet with reason: Maintenance
* 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92310 and previous config saved to /var/cache/conftool/dbconfig/20260505-123411-fceratto.json
* 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 12:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:26 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P92309 and previous config saved to /var/cache/conftool/dbconfig/20260505-122404-fceratto.json
* 12:23 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:23 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P92308 and previous config saved to /var/cache/conftool/dbconfig/20260505-121352-fceratto.json
* 12:04 moritzm: installing postgresql-13 security updates
* 12:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92307 and previous config saved to /var/cache/conftool/dbconfig/20260505-120344-fceratto.json
* 11:57 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282960{{!}}Close Shan Wikinews (T421796)]] (duration: 06m 13s)
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd2005.codfw.wmnet
* 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92306 and previous config saved to /var/cache/conftool/dbconfig/20260505-115535-fceratto.json
* 11:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance
* 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92305 and previous config saved to /var/cache/conftool/dbconfig/20260505-115503-fceratto.json
* 11:53 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 11:53 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282960{{!}}Close Shan Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:52 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd2005.codfw.wmnet
* 11:51 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282960{{!}}Close Shan Wikinews (T421796)]]
* 11:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282955{{!}}Close Norwegian Wikinews (T421796)]] (duration: 09m 21s)
* 11:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd2004.codfw.wmnet
* 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P92304 and previous config saved to /var/cache/conftool/dbconfig/20260505-114455-fceratto.json
* 11:43 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 11:43 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd2004.codfw.wmnet
* 11:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd2003.codfw.wmnet
* 11:39 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282955{{!}}Close Norwegian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd2003.codfw.wmnet
* 11:38 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282955{{!}}Close Norwegian Wikinews (T421796)]]
* 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P92303 and previous config saved to /var/cache/conftool/dbconfig/20260505-113446-fceratto.json
* 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92302 and previous config saved to /var/cache/conftool/dbconfig/20260505-112449-fceratto.json
* 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92301 and previous config saved to /var/cache/conftool/dbconfig/20260505-112438-fceratto.json
* 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92300 and previous config saved to /var/cache/conftool/dbconfig/20260505-111616-fceratto.json
* 11:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance
* 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92299 and previous config saved to /var/cache/conftool/dbconfig/20260505-111545-fceratto.json
* 11:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P92298 and previous config saved to /var/cache/conftool/dbconfig/20260505-111435-fceratto.json
* 11:10 moritzm: installing ca-certificates updates from bookworm point release
* 11:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: after reimage to trixie
* 11:07 moritzm: installing multipart bugfix updates from bookworm point release
* 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P92296 and previous config saved to /var/cache/conftool/dbconfig/20260505-110537-fceratto.json
* 11:05 ayounsi@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 11:05 ayounsi@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P92295 and previous config saved to /var/cache/conftool/dbconfig/20260505-110427-fceratto.json
* 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1174: after reimage to trixie
* 10:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P92293 and previous config saved to /var/cache/conftool/dbconfig/20260505-105529-fceratto.json
* 10:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92291 and previous config saved to /var/cache/conftool/dbconfig/20260505-105419-fceratto.json
* 10:50 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
* 10:50 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
* 10:50 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
* 10:50 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
* 10:49 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
* 10:49 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
* 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92290 and previous config saved to /var/cache/conftool/dbconfig/20260505-104521-fceratto.json
* 10:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92288 and previous config saved to /var/cache/conftool/dbconfig/20260505-104032-fceratto.json
* 10:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92286 and previous config saved to /var/cache/conftool/dbconfig/20260505-103702-fceratto.json
* 10:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance
* 10:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92285 and previous config saved to /var/cache/conftool/dbconfig/20260505-103632-fceratto.json
* 10:32 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 10:29 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P92283 and previous config saved to /var/cache/conftool/dbconfig/20260505-102623-fceratto.json
* 10:24 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:24 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2221: after reimage to trixie
* 10:24 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:23 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:23 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 10:23 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:22 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2221.codfw.wmnet with OS trixie
* 10:17 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 10:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P92281 and previous config saved to /var/cache/conftool/dbconfig/20260505-101616-fceratto.json
* 10:14 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1174: after reimage to trixie
* 09:42 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 09:41 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 09:39 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 09:38 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P92271 and previous config saved to /var/cache/conftool/dbconfig/20260505-093703-fceratto.json
* 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92270 and previous config saved to /var/cache/conftool/dbconfig/20260505-093619-fceratto.json
* 09:36 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 09:35 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92269 and previous config saved to /var/cache/conftool/dbconfig/20260505-093305-fceratto.json
* 09:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 09:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 09:30 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1174.eqiad.wmnet with OS trixie
* 09:30 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2221.codfw.wmnet with OS trixie
* 09:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 09:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:29 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1174: Reimage to Trixie
* 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: Reimage to Trixie
* 09:29 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:28 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1174: Reimage to Trixie
* 09:28 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2221: Reimage to Trixie
* 09:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1174.eqiad.wmnet with reason: Reimage to Trixie
* 09:28 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2221.codfw.wmnet with reason: Reimage to Trixie
* 09:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92265 and previous config saved to /var/cache/conftool/dbconfig/20260505-092654-fceratto.json
* 09:26 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 09:25 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92264 and previous config saved to /var/cache/conftool/dbconfig/20260505-092431-fceratto.json
* 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92263 and previous config saved to /var/cache/conftool/dbconfig/20260505-091808-fceratto.json
* 09:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance
* 09:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P92262 and previous config saved to /var/cache/conftool/dbconfig/20260505-091423-fceratto.json
* 09:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2199.codfw.wmnet with reason: Maintenance
* 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92260 and previous config saved to /var/cache/conftool/dbconfig/20260505-091254-fceratto.json
* 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P92259 and previous config saved to /var/cache/conftool/dbconfig/20260505-090415-fceratto.json
* 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P92258 and previous config saved to /var/cache/conftool/dbconfig/20260505-090246-fceratto.json
* 08:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2209: after reimage to trixie
* 08:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92256 and previous config saved to /var/cache/conftool/dbconfig/20260505-085407-fceratto.json
* 08:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2209.codfw.wmnet with OS trixie
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P92255 and previous config saved to /var/cache/conftool/dbconfig/20260505-085238-fceratto.json
* 08:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 08:50 moritzm: installing augeas security updates
* 08:49 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) install5004.wikimedia.org on all recursors
* 08:48 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 08:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:48 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92254 and previous config saved to /var/cache/conftool/dbconfig/20260505-084616-fceratto.json
* 08:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 08:42 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92253 and previous config saved to /var/cache/conftool/dbconfig/20260505-084231-fceratto.json
* 08:41 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:40 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:38 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:37 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:37 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 08:37 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 08:35 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 08:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 08:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 08:34 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 13 hosts with reason: switches replacement
* 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92252 and previous config saved to /var/cache/conftool/dbconfig/20260505-083356-fceratto.json
* 08:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
* 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92251 and previous config saved to /var/cache/conftool/dbconfig/20260505-083326-fceratto.json
* 08:32 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:32 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 08:32 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 08:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2209.codfw.wmnet with reason: host reimage
* 08:29 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) install5004.wikimedia.org on all recursors
* 08:28 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:28 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:24 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P92250 and previous config saved to /var/cache/conftool/dbconfig/20260505-082318-fceratto.json
* 08:22 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: after reimage to trixie
* 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2209.codfw.wmnet with reason: host reimage
* 08:16 zabe: zabe@deploy1003:~$ foreachwiki refreshImageMetadata --broken-only --mediatype AUDIO --mime audio/x-flac # [[phab:T414641|T414641]]
* 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1170: after reimage to trixie
* 08:14 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P92247 and previous config saved to /var/cache/conftool/dbconfig/20260505-081309-fceratto.json
* 08:08 zabe: zabe@deploy1003:~$ foreachwiki refreshImageMetadata --broken-only --mediatype AUDIO --mime audio/flac # [[phab:T414641|T414641]]
* 08:05 ayounsi@dns1004: END - running authdns-update
* 08:03 ayounsi@dns1004: START - running authdns-update
* 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92245 and previous config saved to /var/cache/conftool/dbconfig/20260505-080301-fceratto.json
* 08:02 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2209.codfw.wmnet with OS trixie
* 08:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ulsfo includes - ayounsi@cumin1003&quot;
* 08:01 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ulsfo includes - ayounsi@cumin1003&quot;
* 08:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: Reimage to Trixie
* 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2209: Reimage to Trixie
* 08:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2209.codfw.wmnet with reason: Reimage to Trixie
* 07:58 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 07:57 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2209 [[phab:T424864|T424864]]', diff saved to https://phabricator.wikimedia.org/P92243 and previous config saved to /var/cache/conftool/dbconfig/20260505-075746-marostegui.json
* 07:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2205 to s3 primary [[phab:T424864|T424864]]', diff saved to https://phabricator.wikimedia.org/P92242 and previous config saved to /var/cache/conftool/dbconfig/20260505-075654-marostegui.json
* 07:55 awight: EU morning deployment was fun
* 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92241 and previous config saved to /var/cache/conftool/dbconfig/20260505-075416-fceratto.json
* 07:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
* 07:52 marostegui: Starting s3 codfw failover from db2209 to db2205 - [[phab:T424864|T424864]]
* 07:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2205 with weight 0 [[phab:T424864|T424864]]', diff saved to https://phabricator.wikimedia.org/P92239 and previous config saved to /var/cache/conftool/dbconfig/20260505-075156-marostegui.json
* 07:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 [[phab:T424864|T424864]]
* 07:50 zabe: zabe@deploy1003:~$ foreachwiki refreshImageMetadata --force --mediatype AUDIO --mime audio/midi # [[phab:T414645|T414645]]
* 07:45 zabe: zabe@deploy1003:~$ mwscript namespaceDupes.php scnwiki --fix # [[phab:T425378|T425378]]
* 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2222: after reimage to trixie
* 07:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2222.codfw.wmnet with OS trixie
* 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1170: after reimage to trixie
* 07:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1170.eqiad.wmnet with OS trixie
* 07:11 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281967{{!}}zhwikinews: (2/2) revert 20th anniversary logo change (assets) (T420165)]] (duration: 06m 43s)
* 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2222.codfw.wmnet with reason: host reimage
* 07:07 awight@deploy1003: awight, 1f616emo: Continuing with deployment
* 07:06 awight@deploy1003: awight, 1f616emo: Backport for [[gerrit:1281967{{!}}zhwikinews: (2/2) revert 20th anniversary logo change (assets) (T420165)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1281967{{!}}zhwikinews: (2/2) revert 20th anniversary logo change (assets) (T420165)]]
* 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1170.eqiad.wmnet with reason: host reimage
* 07:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 07:03 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 07:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2222.codfw.wmnet with reason: host reimage
* 07:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: after reimage to trixie
* 06:58 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 06:58 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 06:58 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1170.eqiad.wmnet with reason: host reimage
* 06:44 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1170.eqiad.wmnet with OS trixie
* 06:44 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2222.codfw.wmnet with OS trixie
* 06:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1170: Reimage to Trixie
* 06:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: Reimage to Trixie
* 06:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1170: Reimage to Trixie
* 06:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Reimage to Trixie
* 06:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2222: Reimage to Trixie
* 06:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2222.codfw.wmnet with reason: Reimage to Trixie
* 06:14 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1156: after reimage to trixie
* 06:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1156.eqiad.wmnet with OS trixie
* 05:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1156.eqiad.wmnet with reason: host reimage
* 05:46 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1156.eqiad.wmnet with reason: host reimage
* 05:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;patterns_as_inline_patterns - oblivian@cumin1003&quot;
* 05:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: patterns_as_inline_patterns - oblivian@cumin1003
* 05:42 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: patterns_as_inline_patterns - oblivian@cumin1003
* 05:42 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;patterns_as_inline_patterns - oblivian@cumin1003&quot;
* 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1156.eqiad.wmnet with OS trixie
* 05:31 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1156: Reimage to Trixie
* 05:30 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1156: Reimage to Trixie
* 05:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Reimage to Trixie
* 05:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: Sanitarium s2 master: reimage to Debian Trixie
* 04:03 mwpresync@deploy1003: Pruned MediaWiki: 1.46.0-wmf.23 (duration: 03m 12s)
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 39s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:21 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 01:21 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add dns infor for new switches - pt1979@cumin2002&quot;
* 01:21 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add dns infor for new switches - pt1979@cumin2002&quot;
* 01:16 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 00:16 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282434{{!}}Close Catalan Wikinews (T421796)]] (duration: 06m 50s)
* 00:11 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 00:10 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282434{{!}}Close Catalan Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:09 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282434{{!}}Close Catalan Wikinews (T421796)]]

== 2026-05-04 ==
* 23:48 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282432{{!}}Close Bosnian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:46 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282432{{!}}Close Bosnian Wikinews (T421796)]]
* 23:14 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282410{{!}}Close Hebrew Wikinews (T421796)]] (duration: 06m 45s)
* 23:10 ladsgroup@deploy1003: neriah, ladsgroup: Continuing with deployment
* 23:09 ladsgroup@deploy1003: neriah, ladsgroup: Backport for [[gerrit:1282410{{!}}Close Hebrew Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282410{{!}}Close Hebrew Wikinews (T421796)]]
* 22:08 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 22:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 22:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 22:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 21:43 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 21:42 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 21:32 cwhite@deploy1003: Finished deploy [statsv/statsv@152de49]: fix logging (duration: 00m 11s)
* 21:32 cwhite@deploy1003: Started deploy [statsv/statsv@152de49]: fix logging
* 21:20 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1276432{{!}}Enable Hebrew keyboard DWIM for namespace resolution on hewikis (T412468)]] (duration: 11m 20s)
* 21:16 cjming@deploy1003: cjming, neriah: Continuing with deployment
* 21:10 cjming@deploy1003: cjming, neriah: Backport for [[gerrit:1276432{{!}}Enable Hebrew keyboard DWIM for namespace resolution on hewikis (T412468)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:09 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1276432{{!}}Enable Hebrew keyboard DWIM for namespace resolution on hewikis (T412468)]]
* 20:38 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282385{{!}}Revert^2 &quot;Use js promise for email confirmation banner&quot;]] (duration: 22m 19s)
* 20:34 cjming@deploy1003: mmartorana, cjming: Continuing with deployment
* 20:18 cjming@deploy1003: mmartorana, cjming: Backport for [[gerrit:1282385{{!}}Revert^2 &quot;Use js promise for email confirmation banner&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:16 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1282385{{!}}Revert^2 &quot;Use js promise for email confirmation banner&quot;]]
* 20:11 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277667{{!}}Enable the reading list beta feature survey on all wikipedias (T421776)]] (duration: 07m 21s)
* 20:07 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1005.eqiad.wmnet with OS trixie
* 20:06 toyofuku@deploy1003: toyofuku: Continuing with deployment
* 20:05 toyofuku@deploy1003: toyofuku: Backport for [[gerrit:1277667{{!}}Enable the reading list beta feature survey on all wikipedias (T421776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1277667{{!}}Enable the reading list beta feature survey on all wikipedias (T421776)]]
* 19:51 ayounsi@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) asw1-22-ulsfo.wikimedia.org on all recursors
* 19:50 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache asw1-22-ulsfo.wikimedia.org on all recursors
* 19:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: asw1-22-ulsfo - ayounsi@cumin1003&quot;
* 19:49 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: asw1-22-ulsfo - ayounsi@cumin1003&quot;
* 19:48 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1005.eqiad.wmnet with reason: host reimage
* 19:44 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 19:42 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1005.eqiad.wmnet with reason: host reimage
* 19:40 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:37 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:28 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: ongoing troubleshooting
* 19:27 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1005
* 19:27 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1005
* 19:27 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1005.eqiad.wmnet with OS trixie
* 19:23 root@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
* 19:23 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:23 root@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
* 19:23 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 19:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 18:59 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282407{{!}}Close Limburgish Wikinews (T421796)]] (duration: 06m 16s)
* 18:55 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 18:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282407{{!}}Close Limburgish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:53 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282407{{!}}Close Limburgish Wikinews (T421796)]]
* 18:31 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282405{{!}}Close Albanian Wikinews (T421796)]] (duration: 09m 17s)
* 18:27 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 18:23 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282405{{!}}Close Albanian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:22 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282405{{!}}Close Albanian Wikinews (T421796)]]
* 18:11 dancy@deploy1003: Finished scap sync-world: testing (duration: 02m 04s)
* 18:11 dancy@deploy1003: dancy: Rolling back deployment
* 18:10 dancy@deploy1003: dancy: testing synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:09 dancy@deploy1003: Started scap sync-world: testing
* 18:08 dancy@deploy1003: Installation of scap version &quot;4.260.0&quot; completed for 2 hosts
* 18:06 dancy@deploy1003: Installing scap version &quot;4.260.0&quot; for 2 host(s)
* 17:49 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:47 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:47 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:41 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:31 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 16:40 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:39 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:34 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 16:33 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:33 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 16:33 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:04 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282384{{!}}Close Greek Wikinews (T421796)]] (duration: 06m 19s)
* 16:00 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 16:00 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282384{{!}}Close Greek Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:58 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282384{{!}}Close Greek Wikinews (T421796)]]
* 15:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92224 and previous config saved to /var/cache/conftool/dbconfig/20260504-155514-fceratto.json
* 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92223 and previous config saved to /var/cache/conftool/dbconfig/20260504-154506-fceratto.json
* 15:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282060{{!}}Make errorpages responsive]] (duration: 06m 59s)
* 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92222 and previous config saved to /var/cache/conftool/dbconfig/20260504-153458-fceratto.json
* 15:34 ladsgroup@deploy1003: ladsgroup, chlod: Continuing with deployment
* 15:33 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 39 hosts with reason: switches replacement
* 15:33 ladsgroup@deploy1003: ladsgroup, chlod: Backport for [[gerrit:1282060{{!}}Make errorpages responsive]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:32 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: sync
* 15:32 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: sync
* 15:31 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282060{{!}}Make errorpages responsive]]
* 15:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92221 and previous config saved to /var/cache/conftool/dbconfig/20260504-152449-fceratto.json
* 15:22 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92220 and previous config saved to /var/cache/conftool/dbconfig/20260504-152238-fceratto.json
* 15:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance
* 15:20 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:17 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 15:17 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 15:16 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:15 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:13 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 15:13 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92219 and previous config saved to /var/cache/conftool/dbconfig/20260504-151238-fceratto.json
* 15:10 papaul: ongoing switch refresh in ULSFO
* 15:10 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 15:10 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 15:06 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:05 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282381{{!}}Close Gun Wikinews (T421796)]] (duration: 06m 45s)
* 15:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92218 and previous config saved to /var/cache/conftool/dbconfig/20260504-150230-fceratto.json
* 15:01 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:00 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282381{{!}}Close Gun Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:58 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282381{{!}}Close Gun Wikinews (T421796)]]
* 14:58 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2001.codfw.wmnet with OS trixie
* 14:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92217 and previous config saved to /var/cache/conftool/dbconfig/20260504-145222-fceratto.json
* 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92216 and previous config saved to /var/cache/conftool/dbconfig/20260504-144213-fceratto.json
* 14:41 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts
* 14:41 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 7 hosts
* 14:39 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2001.codfw.wmnet with reason: host reimage
* 14:34 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2001.codfw.wmnet with reason: host reimage
* 14:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92215 and previous config saved to /var/cache/conftool/dbconfig/20260504-143334-fceratto.json
* 14:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance
* 14:30 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cr[3-4]-ulsfo IPv6,cr[3-4]-ulsfo.mgmt,mr1-ulsfo IPv6 with reason: switch refresh
* 14:28 pt1979@cumin1003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 4:00:00 on cr[3-4]-ulsfo IPV6,cr[3-4]-ulsfo.mgmt,mr1-ulsfo IPV6 with reason: switch refresh
* 14:25 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on asw2-ulsfo,cr[3-4]-ulsfo,mr1-ulsfo with reason: switch refresh
* 14:16 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2001
* 14:16 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2001
* 14:13 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2001
* 14:13 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2001.codfw.wmnet 94.0.192.10.in-addr.arpa 4.9.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:13 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2001.codfw.wmnet 94.0.192.10.in-addr.arpa 4.9.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:13 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:13 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2001 - herron@cumin1003&quot;
* 14:13 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2001 - herron@cumin1003&quot;
* 14:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92214 and previous config saved to /var/cache/conftool/dbconfig/20260504-141113-fceratto.json
* 14:07 herron@cumin1003: START - Cookbook sre.dns.netbox
* 14:04 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2001
* 14:04 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2001.codfw.wmnet with OS trixie
* 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P92213 and previous config saved to /var/cache/conftool/dbconfig/20260504-140105-fceratto.json
* 14:00 slyngshede@cumin1003: conftool action : set/pooled=no; selector: cluster=dnsbox,dc=ulsfo [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 14:00 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 14:00 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 13:59 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282354{{!}}ArticleGuidance: enable on simple english (T425351)]] (duration: 06m 22s)
* 13:57 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 13:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 13:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:56 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 13:56 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 13:55 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 13:55 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 13:55 sbisson@deploy1003: sbisson: Continuing with deployment
* 13:55 sbisson@deploy1003: sbisson: Backport for [[gerrit:1282354{{!}}ArticleGuidance: enable on simple english (T425351)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:54 dcausse: [[phab:T425301|T425301]]: stopping writes again on cloudelastic, cluster unstable
* 13:53 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1282354{{!}}ArticleGuidance: enable on simple english (T425351)]]
* 13:52 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:52 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P92212 and previous config saved to /var/cache/conftool/dbconfig/20260504-135056-fceratto.json
* 13:50 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281965{{!}}zhwikinews: (1/2) revert 20th anniversary logo change (config) (T420165)]] (duration: 07m 30s)
* 13:46 sbisson@deploy1003: 1f616emo, sbisson: Continuing with deployment
* 13:45 sbisson@deploy1003: 1f616emo, sbisson: Backport for [[gerrit:1281965{{!}}zhwikinews: (1/2) revert 20th anniversary logo change (config) (T420165)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1281965{{!}}zhwikinews: (1/2) revert 20th anniversary logo change (config) (T420165)]]
* 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92211 and previous config saved to /var/cache/conftool/dbconfig/20260504-134048-fceratto.json
* 13:40 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92210 and previous config saved to /var/cache/conftool/dbconfig/20260504-133039-fceratto.json
* 13:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92209 and previous config saved to /var/cache/conftool/dbconfig/20260504-133010-fceratto.json
* 13:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:23 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:23 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P92208 and previous config saved to /var/cache/conftool/dbconfig/20260504-132002-fceratto.json
* 13:13 moritzm: installing jaraco.context security updates
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum5004.eqsin.wmnet
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host durum5004.eqsin.wmnet with OS bookworm
* 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P92207 and previous config saved to /var/cache/conftool/dbconfig/20260504-130953-fceratto.json
* 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92206 and previous config saved to /var/cache/conftool/dbconfig/20260504-125945-fceratto.json
* 12:59 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 12:59 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 12:59 dcausse: [[phab:T425301|T425301]]: resuming writes on cloudelastic
* 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92205 and previous config saved to /var/cache/conftool/dbconfig/20260504-125247-fceratto.json
* 12:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92204 and previous config saved to /var/cache/conftool/dbconfig/20260504-125219-fceratto.json
* 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 12:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on durum5004.eqsin.wmnet with reason: host reimage
* 12:45 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on durum5004.eqsin.wmnet with reason: host reimage
* 12:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P92203 and previous config saved to /var/cache/conftool/dbconfig/20260504-124210-fceratto.json
* 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P92202 and previous config saved to /var/cache/conftool/dbconfig/20260504-123203-fceratto.json
* 12:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92201 and previous config saved to /var/cache/conftool/dbconfig/20260504-122155-fceratto.json
* 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1203 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92200 and previous config saved to /var/cache/conftool/dbconfig/20260504-121441-fceratto.json
* 12:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
* 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92199 and previous config saved to /var/cache/conftool/dbconfig/20260504-121424-fceratto.json
* 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P92198 and previous config saved to /var/cache/conftool/dbconfig/20260504-120416-fceratto.json
* 12:03 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host durum5004.eqsin.wmnet with OS bookworm
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) durum5004.eqsin.wmnet on all recursors
* 11:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache durum5004.eqsin.wmnet on all recursors
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P92197 and previous config saved to /var/cache/conftool/dbconfig/20260504-115408-fceratto.json
* 11:51 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:47 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 11:47 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host durum5004.eqsin.wmnet
* 11:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum5003.eqsin.wmnet
* 11:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host durum5003.eqsin.wmnet with OS bookworm
* 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92196 and previous config saved to /var/cache/conftool/dbconfig/20260504-114400-fceratto.json
* 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1193 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92195 and previous config saved to /var/cache/conftool/dbconfig/20260504-113620-fceratto.json
* 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
* 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92194 and previous config saved to /var/cache/conftool/dbconfig/20260504-113550-fceratto.json
* 11:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: after reimage to trixie
* 11:26 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on durum5003.eqsin.wmnet with reason: host reimage
* 11:25 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on durum5003.eqsin.wmnet with reason: host reimage
* 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P92192 and previous config saved to /var/cache/conftool/dbconfig/20260504-112542-fceratto.json
* 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P92191 and previous config saved to /var/cache/conftool/dbconfig/20260504-111534-fceratto.json
* 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92189 and previous config saved to /var/cache/conftool/dbconfig/20260504-110526-fceratto.json
* 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2187: repool after maintenance
* 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92187 and previous config saved to /var/cache/conftool/dbconfig/20260504-105808-fceratto.json
* 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92186 and previous config saved to /var/cache/conftool/dbconfig/20260504-105739-fceratto.json
* 10:48 moritzm: installing bash updates from trixie point release
* 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P92184 and previous config saved to /var/cache/conftool/dbconfig/20260504-104731-fceratto.json
* 10:42 moritzm: installing postgresql-17 security updates
* 10:42 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1162: after reimage to trixie
* 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1162.eqiad.wmnet with OS trixie
* 10:38 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS bookworm
* 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P92181 and previous config saved to /var/cache/conftool/dbconfig/20260504-103723-fceratto.json
* 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:35 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) durum5003.eqsin.wmnet on all recursors
* 10:34 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache durum5003.eqsin.wmnet on all recursors
* 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:34 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92179 and previous config saved to /var/cache/conftool/dbconfig/20260504-102715-fceratto.json
* 10:26 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host durum5003.eqsin.wmnet
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1178 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92178 and previous config saved to /var/cache/conftool/dbconfig/20260504-101855-fceratto.json
* 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92177 and previous config saved to /var/cache/conftool/dbconfig/20260504-101826-fceratto.json
* 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2187: repool after maintenance
* 10:16 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
* 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
* 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P92174 and previous config saved to /var/cache/conftool/dbconfig/20260504-100818-fceratto.json
* 10:02 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1162.eqiad.wmnet with OS trixie
* 10:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reimage to Trixie
* 10:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reimage to Trixie
* 10:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Reimage to Trixie
* 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P92172 and previous config saved to /var/cache/conftool/dbconfig/20260504-095810-fceratto.json
* 09:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org
* 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92171 and previous config saved to /var/cache/conftool/dbconfig/20260504-094802-fceratto.json
* 09:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1177 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92170 and previous config saved to /var/cache/conftool/dbconfig/20260504-093938-fceratto.json
* 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92169 and previous config saved to /var/cache/conftool/dbconfig/20260504-093910-fceratto.json
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1182: after reimage to trixie
* 09:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P92167 and previous config saved to /var/cache/conftool/dbconfig/20260504-092902-fceratto.json
* 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P92165 and previous config saved to /var/cache/conftool/dbconfig/20260504-091853-fceratto.json
* 09:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2187: Fixing events
* 09:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2187: Fixing events
* 09:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2187.codfw.wmnet with reason: Checking events
* 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92163 and previous config saved to /var/cache/conftool/dbconfig/20260504-090845-fceratto.json
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92161 and previous config saved to /var/cache/conftool/dbconfig/20260504-085930-fceratto.json
* 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92160 and previous config saved to /var/cache/conftool/dbconfig/20260504-085912-fceratto.json
* 08:56 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 08:55 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 08:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1182: after reimage to trixie
* 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P92158 and previous config saved to /var/cache/conftool/dbconfig/20260504-084904-fceratto.json
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1008.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1007.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1006.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1005.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1004.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1003.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1002.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1001.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-ctrl1002.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-ctrl1001.eqiad.wmnet
* 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P92157 and previous config saved to /var/cache/conftool/dbconfig/20260504-083857-fceratto.json
* 08:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1182.eqiad.wmnet with OS trixie
* 08:32 moritzm: installing Linux 5.10.251-3 on bullseye hosts
* 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92156 and previous config saved to /var/cache/conftool/dbconfig/20260504-082849-fceratto.json
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1003.eqiad.wmnet
* 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92155 and previous config saved to /var/cache/conftool/dbconfig/20260504-082024-fceratto.json
* 08:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 08:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 08:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1003.eqiad.wmnet
* 08:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1182.eqiad.wmnet with reason: host reimage
* 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1182.eqiad.wmnet with reason: host reimage
* 08:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
* 08:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
* 08:04 gkyziridis@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync
* 08:04 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277256{{!}}Add sva to wmgExtraLanguageNames (T407106)]] (duration: 07m 58s)
* 08:03 gkyziridis@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync
* 08:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 08:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 08:02 gkyziridis@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: sync
* 08:02 gkyziridis@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: sync
* 08:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
* 08:01 moritzm: installing Linux 6.1.170 on bookworm hosts
* 07:59 urbanecm@deploy1003: urbanecm, h2o: Continuing with deployment
* 07:57 urbanecm@deploy1003: urbanecm, h2o: Backport for [[gerrit:1277256{{!}}Add sva to wmgExtraLanguageNames (T407106)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1212: after reimage to trixie
* 07:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1277256{{!}}Add sva to wmgExtraLanguageNames (T407106)]]
* 07:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1182.eqiad.wmnet with OS trixie
* 07:51 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 07:51 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 07:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1182: Reimage to Trixie
* 07:47 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1182: Reimage to Trixie
* 07:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Reimage to Trixie
* 07:44 dcausse: [[phab:T425301|T425301]]: stopping writes on cloudelastic
* 07:44 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:44 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2147.codfw.wmnet
* 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2147.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 07:42 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2147.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2149: after reimage to trixie
* 07:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1188: after reimage to trixie
* 07:38 moritzm: installing Linux 6.12.85 on trixie hosts
* 07:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-jumbo2003.codfw.wmnet
* 07:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-jumbo2003.codfw.wmnet
* 07:33 marostegui@cumin1003: START - Cookbook sre.dns.netbox
* 07:28 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2147.codfw.wmnet
* 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org
* 07:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test1005.wikimedia.org
* 07:11 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1212: after reimage to trixie
* 07:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1212.eqiad.wmnet with OS trixie
* 06:56 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2149: after reimage to trixie
* 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1188: after reimage to trixie
* 06:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1188.eqiad.wmnet with OS trixie
* 06:47 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2149.codfw.wmnet with OS trixie
* 06:43 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1212.eqiad.wmnet with reason: host reimage
* 06:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1212.eqiad.wmnet with reason: host reimage
* 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
* 06:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
* 06:21 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1212.eqiad.wmnet with OS trixie
* 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
* 06:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
* 06:11 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1212: Reimage to Trixie
* 06:11 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1212: Reimage to Trixie
* 06:11 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1212.eqiad.wmnet with reason: Reimage to Trixie
* 06:10 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: Sanitarium s3 master: reimage to Debian Trixie
* 06:09 marostegui: Reimage sanitarium master for s3, lag to be expected on wikireplicas for s3 [[phab:T424792|T424792]]
* 06:05 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1188.eqiad.wmnet with OS trixie
* 06:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1188: Reimage to Trixie
* 05:58 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1188: Reimage to Trixie
* 05:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Reimage to Trixie
* 05:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2149.codfw.wmnet with OS trixie
* 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2149: Reimage to Trixie
* 05:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2149: Reimage to Trixie
* 05:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2149.codfw.wmnet with reason: Reimage to Trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 36s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-05-03 ==
* 14:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281991{{!}}Disable uploads in scnwiki (T425278)]] (duration: 10m 51s)
* 14:05 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 14:04 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1281991{{!}}Disable uploads in scnwiki (T425278)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:00 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1281991{{!}}Disable uploads in scnwiki (T425278)]]
* 12:27 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281963{{!}}Remove Wikinews from installer's default main page]] (duration: 29m 22s)
* 11:58 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1281963{{!}}Remove Wikinews from installer's default main page]]

== 2026-05-02 ==
* 23:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281775{{!}}Uninstall DynamicPageList from wikis it's not used on (T425202)]] (duration: 06m 41s)
* 23:28 zabe@deploy1003: dreamyjazz, zabe: Continuing with deployment
* 23:27 zabe@deploy1003: dreamyjazz, zabe: Backport for [[gerrit:1281775{{!}}Uninstall DynamicPageList from wikis it's not used on (T425202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:26 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281775{{!}}Uninstall DynamicPageList from wikis it's not used on (T425202)]]
* 23:22 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281739{{!}}Uninstall DynamicPageList from officewiki (T425154)]] (duration: 07m 27s)
* 23:18 zabe@deploy1003: zabe, dreamyjazz: Continuing with deployment
* 23:17 zabe@deploy1003: zabe, dreamyjazz: Backport for [[gerrit:1281739{{!}}Uninstall DynamicPageList from officewiki (T425154)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281739{{!}}Uninstall DynamicPageList from officewiki (T425154)]]
* 18:07 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie
* 18:07 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie
* 18:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host rdb2014.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host rdb2014.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:43 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2369.codfw.wmnet with OS trixie
* 17:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:36 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:18 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2369.codfw.wmnet with reason: host reimage
* 17:14 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2369.codfw.wmnet with reason: host reimage
* 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2374.codfw.wmnet with OS trixie
* 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:13 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:09 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2373.codfw.wmnet with OS trixie
* 17:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:09 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2372.codfw.wmnet with OS trixie
* 17:06 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:05 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:01 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2371.codfw.wmnet with OS trixie
* 17:01 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2370.codfw.wmnet with OS trixie
* 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 16:56 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2374.codfw.wmnet with reason: host reimage
* 16:56 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 16:52 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2373.codfw.wmnet with reason: host reimage
* 16:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 16:48 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2372.codfw.wmnet with reason: host reimage
* 16:44 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2374.codfw.wmnet with reason: host reimage
* 16:43 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2373.codfw.wmnet with reason: host reimage
* 16:42 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2372.codfw.wmnet with reason: host reimage
* 16:36 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2371.codfw.wmnet with reason: host reimage
* 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2370.codfw.wmnet with reason: host reimage
* 16:32 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2371.codfw.wmnet with reason: host reimage
* 16:31 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2374.codfw.wmnet with OS trixie
* 16:30 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2373.codfw.wmnet with OS trixie
* 16:29 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2372.codfw.wmnet with OS trixie
* 16:28 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2370.codfw.wmnet with reason: host reimage
* 16:20 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2371.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2371.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2371.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2370.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2369.codfw.wmnet with OS trixie
* 16:08 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2366.codfw.wmnet with OS trixie
* 16:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2367.codfw.wmnet with OS trixie
* 15:59 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2368.codfw.wmnet with OS trixie
* 15:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 15:47 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 15:43 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 15:38 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 15:38 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 15:36 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 15:25 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2366.codfw.wmnet with OS trixie
* 15:24 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2367.codfw.wmnet with OS trixie
* 15:23 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2368.codfw.wmnet with OS trixie
* 12:02 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281747{{!}}Watchlist star: Revert popover/dialog changes (T425185)]] (duration: 13m 06s)
* 11:57 samtar@deploy1003: samtar: Continuing with deployment
* 11:50 samtar@deploy1003: samtar: Backport for [[gerrit:1281747{{!}}Watchlist star: Revert popover/dialog changes (T425185)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:49 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1281747{{!}}Watchlist star: Revert popover/dialog changes (T425185)]]
* 09:20 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2366.codfw.wmnet with OS trixie
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2368.codfw.wmnet with OS trixie
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2367.codfw.wmnet with OS trixie
* 02:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:23 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:17 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:12 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 31s)
* 02:02 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:58 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 01:54 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 01:50 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 01:50 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 01:49 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 01:37 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2368.codfw.wmnet with OS trixie
* 01:37 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2367.codfw.wmnet with OS trixie
* 01:37 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2366.codfw.wmnet with OS trixie
* 01:35 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2364.codfw.wmnet with OS trixie
* 01:34 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:29 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:24 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2365.codfw.wmnet with OS trixie
* 01:24 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:23 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:21 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2363.codfw.wmnet with OS trixie
* 01:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:13 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2364.codfw.wmnet with reason: host reimage
* 01:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2365.codfw.wmnet with reason: host reimage
* 01:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2363.codfw.wmnet with reason: host reimage
* 00:58 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2365.codfw.wmnet with reason: host reimage
* 00:57 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2364.codfw.wmnet with reason: host reimage
* 00:57 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2363.codfw.wmnet with reason: host reimage
* 00:45 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2365.codfw.wmnet with OS trixie
* 00:45 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2364.codfw.wmnet with OS trixie
* 00:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2363.codfw.wmnet with OS trixie
* 00:09 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2362.codfw.wmnet with OS trixie
* 00:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:07 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2361.codfw.wmnet with OS trixie
* 00:05 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:02 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:01 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2360.codfw.wmnet with OS trixie
* 00:01 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:01 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;

== 2026-05-01 ==
* 23:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2362.codfw.wmnet with reason: host reimage
* 23:47 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2361.codfw.wmnet with reason: host reimage
* 23:45 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2360.codfw.wmnet with reason: host reimage
* 23:39 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2362.codfw.wmnet with reason: host reimage
* 23:39 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2361.codfw.wmnet with reason: host reimage
* 23:39 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2360.codfw.wmnet with reason: host reimage
* 23:27 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2362.codfw.wmnet with OS trixie
* 23:27 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2361.codfw.wmnet with OS trixie
* 23:26 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2360.codfw.wmnet with OS trixie
* 23:25 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2357.codfw.wmnet with OS trixie
* 23:25 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:25 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:22 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2359.codfw.wmnet with OS trixie
* 23:22 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:22 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:18 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2358.codfw.wmnet with OS trixie
* 23:18 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:18 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:08 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2357.codfw.wmnet with reason: host reimage
* 23:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2359.codfw.wmnet with reason: host reimage
* 23:02 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2358.codfw.wmnet with reason: host reimage
* 22:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2359.codfw.wmnet with reason: host reimage
* 22:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2357.codfw.wmnet with reason: host reimage
* 22:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2358.codfw.wmnet with reason: host reimage
* 22:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2359.codfw.wmnet with OS trixie
* 22:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2358.codfw.wmnet with OS trixie
* 22:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2357.codfw.wmnet with OS trixie
* 22:39 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:28 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:25 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:24 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:19 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:17 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:15 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:06 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:02 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:59 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:57 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:51 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:49 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:48 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:48 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:46 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:41 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:41 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2364.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:38 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2363.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:33 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2364.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:32 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:31 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2363.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:25 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2361.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:24 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2360.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:21 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2361.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:17 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2360.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:15 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2359.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:13 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2358.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2359.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2358.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:04 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2374
* 21:04 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2374
* 21:04 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2373
* 21:04 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2373
* 20:59 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2372
* 20:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2372
* 20:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2371
* 20:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2371
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2370
* 20:57 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2370
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2369
* 20:57 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2369
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2368
* 20:57 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2368
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2367
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2367
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2366
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2366
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2365
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2365
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2364
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2364
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2363
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2363
* 20:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2362
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2362
* 20:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2361
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2361
* 20:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2360
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2360
* 20:54 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2359
* 20:54 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2359
* 20:54 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2358
* 20:54 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2358
* 20:54 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2357
* 20:53 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2357
* 20:53 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:53 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2357 to codfw - jhancock@cumin2002&quot;
* 20:53 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2357 to codfw - jhancock@cumin2002&quot;
* 20:49 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 20:14 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2002.codfw.wmnet with OS trixie
* 20:06 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269440{{!}}Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441{{!}}Enable wgTrackMediaRequestProvenance on Commons (T414338)]] (duration: 15m 27s)
* 20:02 krinkle@deploy1003: krinkle: Continuing with deployment
* 19:54 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage
* 19:52 krinkle@deploy1003: krinkle: Backport for [[gerrit:1269440{{!}}Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441{{!}}Enable wgTrackMediaRequestProvenance on Commons (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:51 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1269440{{!}}Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441{{!}}Enable wgTrackMediaRequestProvenance on Commons (T414338)]]
* 19:49 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage
* 19:40 dancy@deploy1003: Finished scap sync-world: testing [[phab:T317405|T317405]] (duration: 03m 23s)
* 19:37 dancy@deploy1003: Started scap sync-world: testing [[phab:T317405|T317405]]
* 19:36 dancy@deploy1003: Installation of scap version &quot;4.259.0&quot; completed for 2 hosts
* 19:34 dancy@deploy1003: Installing scap version &quot;4.259.0&quot; for 2 host(s)
* 18:55 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 18:55 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 18:43 jhathaway@cumin1003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Alangi Derick out of all services on: 2442 hosts
* 18:41 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2002
* 18:41 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2002
* 18:41 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2002
* 18:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2002.codfw.wmnet 50.16.192.10.in-addr.arpa 0.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:40 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2002.codfw.wmnet 50.16.192.10.in-addr.arpa 0.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:40 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2002 - herron@cumin1003&quot;
* 18:40 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2002 - herron@cumin1003&quot;
* 18:36 herron@cumin1003: START - Cookbook sre.dns.netbox
* 18:33 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2002
* 18:32 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2002.codfw.wmnet with OS trixie
* 18:26 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2003.codfw.wmnet with OS trixie
* 18:04 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage
* 18:00 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage
* 17:41 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2003
* 17:41 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2003
* 17:40 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2003
* 17:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2003.codfw.wmnet 24.32.192.10.in-addr.arpa 4.2.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:40 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2003.codfw.wmnet 24.32.192.10.in-addr.arpa 4.2.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:40 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2003 - herron@cumin1003&quot;
* 17:40 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2003 - herron@cumin1003&quot;
* 17:33 herron@cumin1003: START - Cookbook sre.dns.netbox
* 17:28 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2003
* 17:28 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2003.codfw.wmnet with OS trixie
* 17:15 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2004.codfw.wmnet with OS trixie
* 16:34 cdobbins@cumin2002: conftool action : get/pooled; selector: name=cp5024.eqsin.wmnet
* 16:30 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 16:30 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 16:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
* 16:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
* 16:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2001.codfw.wmnet
* 15:59 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 15:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest2001.codfw.wmnet
* 15:47 dancy@deploy1003: Installation of scap version &quot;4.258.1&quot; completed for 2 hosts
* 15:45 dancy@deploy1003: Installing scap version &quot;4.258.1&quot; for 2 host(s)
* 15:34 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2004.codfw.wmnet with reason: host reimage
* 15:30 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2004.codfw.wmnet with reason: host reimage
* 15:14 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2004
* 15:14 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2004
* 15:11 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2004
* 15:11 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2004.codfw.wmnet 38.16.192.10.in-addr.arpa 8.3.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:11 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2004.codfw.wmnet 38.16.192.10.in-addr.arpa 8.3.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:11 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2004 - herron@cumin1003&quot;
* 15:11 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2004 - herron@cumin1003&quot;
* 15:05 dancy@deploy1003: Installation of scap version &quot;4.258.0&quot; completed for 2 hosts
* 15:03 dancy@deploy1003: Installing scap version &quot;4.258.0&quot; for 2 host(s)
* 14:57 herron@cumin1003: START - Cookbook sre.dns.netbox
* 14:47 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2004
* 14:47 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2004.codfw.wmnet with OS trixie
* 13:45 zabe@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 13:44 zabe@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 13:24 _Gerges: WikiMonitor setup
* 13:09 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080
* 13:09 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078
* 13:09 jclark@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1079
* 13:09 jclark@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1077
* 13:09 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080
* 13:09 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079
* 13:09 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078
* 13:08 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077
* 13:08 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:08 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:04 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:04 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1077 to eqiad - jclark@cumin1003&quot;
* 13:04 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1077 to eqiad - jclark@cumin1003&quot;
* 13:00 jclark@cumin1003: START - Cookbook sre.dns.netbox
* 12:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 12:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 12:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 12:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 09:57 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281423{{!}}Switch watchstar from Popover to Dialog (T417847)]] (duration: 06m 49s)
* 09:53 samtar@deploy1003: samtar: Continuing with deployment
* 09:52 samtar@deploy1003: samtar: Backport for [[gerrit:1281423{{!}}Switch watchstar from Popover to Dialog (T417847)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:50 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1281423{{!}}Switch watchstar from Popover to Dialog (T417847)]]
* 09:38 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281426{{!}}Update the interwiki cache (T239173)]] (duration: 06m 05s)
* 09:32 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1281426{{!}}Update the interwiki cache (T239173)]]
* 08:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 08:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 03:26 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 03:26 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 00:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1280417{{!}}Add script to fix fr_deleted drifts (T424553)]] (duration: 07m 05s)
* 00:13 zabe@deploy1003: zabe: Continuing with deployment
* 00:11 zabe@deploy1003: zabe: Backport for [[gerrit:1280417{{!}}Add script to fix fr_deleted drifts (T424553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1280417{{!}}Add script to fix fr_deleted drifts (T424553)]]

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>14ic4mzfvr5dd5d5pdcmz9235u7krsi</sha1>
    </revision>
    <revision>
      <id>2412194</id>
      <parentid>2412193</parentid>
      <timestamp>2026-05-10T02:06:56Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 36s)</comment>
      <origin>2412194</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="311760" sha1="2ahfzbccjfs6frtgb0ud6v0xansl8s8" xml:space="preserve">
== 2026-05-10 ==
* 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 36s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-05-09 ==
* 10:34 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix dsl column size - oblivian@cumin1003&quot;
* 10:34 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix dsl column size - oblivian@cumin1003
* 10:33 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix dsl column size - oblivian@cumin1003
* 10:33 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix dsl column size - oblivian@cumin1003&quot;
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s)
* 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:06 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1267.eqiad.wmnet with OS bookworm
* 01:06 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 01:06 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 00:49 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1267.eqiad.wmnet with reason: host reimage
* 00:44 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1267.eqiad.wmnet with reason: host reimage
* 00:29 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1267.eqiad.wmnet with OS bookworm
* 00:17 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1267.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED

== 2026-05-08 ==
* 23:55 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1267.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:35 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1267
* 23:32 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1267
* 23:30 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 23:30 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1267] - vriley@cumin1003&quot;
* 23:30 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1267] - vriley@cumin1003&quot;
* 23:26 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 23:22 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1266.eqiad.wmnet with OS bookworm
* 23:22 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 23:11 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 22:54 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1266.eqiad.wmnet with reason: host reimage
* 22:46 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1266.eqiad.wmnet with reason: host reimage
* 22:26 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1266.eqiad.wmnet with OS bookworm
* 22:16 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1266.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:56 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1266.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:55 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1266
* 21:53 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1266
* 21:52 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:52 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1266] - vriley@cumin1003&quot;
* 21:51 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1266] - vriley@cumin1003&quot;
* 21:45 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 21:42 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS bookworm
* 21:42 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 21:41 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 21:24 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage
* 21:19 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage
* 20:54 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS bookworm
* 20:44 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1265.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:32 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1265.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:31 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1265
* 20:30 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1265
* 20:29 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:29 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1265] - vriley@cumin1003&quot;
* 20:29 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [db1265] - vriley@cumin1003&quot;
* 20:24 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 20:01 ryankemper: [WDQS] Added several more requestctl rules. They've helped marginally, but not enough to restore the service. Unless we find an obvious smoking gun, expect noise to continue for the timebeing :/
* 19:42 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 19:41 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 19:41 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 19:40 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 18:07 ryankemper: [WDQS] After those 2 requestctl rules, requests went down 20%, error rate decreased significantly, p50 cut almost in half, but the service is still unstable, likely we'll need to identify more throttle-candidates to restore full health
* 17:53 ryankemper: [WDQS] Deployed 2 new requestctl rules; we'll see if it helps
* 16:51 topranks: enable bfd on system0.0 sub-interface ssw1-d1-eqiad
* 15:45 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on ms-backup1003.eqiad.wmnet with reason: restart
* 15:37 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup[1006,1017-1018].eqiad.wmnet with reason: restart
* 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-jumbo1001.eqiad.wmnet
* 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-jumbo1001.eqiad.wmnet
* 14:07 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 10:51 btullis: re-pooled wdqs-main in eqiad for [[phab:T425758|T425758]]
* 10:50 btullis@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-main,name=eqiad
* 10:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:15 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:14 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup1007.eqiad.wmnet with reason: restart
* 10:12 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:11 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:11 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:09 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:09 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:44 btullis: depooled wdqs-main in eqiad for [[phab:T425758|T425758]]
* 09:41 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 09:41 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:41 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:41 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:40 btullis@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=wdqs-main,name=eqiad
* 09:36 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 09:36 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:36 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:35 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92437 and previous config saved to /var/cache/conftool/dbconfig/20260508-093251-fceratto.json
* 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P92435 and previous config saved to /var/cache/conftool/dbconfig/20260508-092243-fceratto.json
* 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P92434 and previous config saved to /var/cache/conftool/dbconfig/20260508-091238-fceratto.json
* 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92433 and previous config saved to /var/cache/conftool/dbconfig/20260508-090230-fceratto.json
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92432 and previous config saved to /var/cache/conftool/dbconfig/20260508-085217-fceratto.json
* 08:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
* 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92431 and previous config saved to /var/cache/conftool/dbconfig/20260508-085018-fceratto.json
* 08:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92430 and previous config saved to /var/cache/conftool/dbconfig/20260508-084010-fceratto.json
* 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92429 and previous config saved to /var/cache/conftool/dbconfig/20260508-083003-fceratto.json
* 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92428 and previous config saved to /var/cache/conftool/dbconfig/20260508-081954-fceratto.json
* 08:18 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 08:17 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92427 and previous config saved to /var/cache/conftool/dbconfig/20260508-080438-fceratto.json
* 08:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 07:56 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts install5003.wikimedia.org
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: install5003.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: install5003.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 07:09 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2159: after reimage to trixie
* 06:57 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts install5003.wikimedia.org
* 06:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2159: after reimage to trixie
* 06:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2159.codfw.wmnet with OS trixie
* 06:11 moritzm: installing postorius security updates
* 05:51 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2159.codfw.wmnet with reason: host reimage
* 05:46 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2159.codfw.wmnet with reason: host reimage
* 05:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2159.codfw.wmnet with OS trixie
* 05:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2159: Reimage to Trixie
* 05:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2159: Reimage to Trixie
* 05:25 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2159.codfw.wmnet with reason: Reimage to Trixie
* 03:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1024.eqiad.wmnet with OS trixie
* 03:04 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 03:04 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 02:50 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1024.eqiad.wmnet with reason: host reimage
* 02:45 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1024.eqiad.wmnet with reason: host reimage
* 02:32 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1024.eqiad.wmnet with OS trixie
* 02:23 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1024.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 02:07 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1024.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 02:07 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1024
* 02:05 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1024
* 02:04 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 02:04 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1024] - vriley@cumin1003&quot;
* 02:04 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1024] - vriley@cumin1003&quot;
* 02:01 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 01:52 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1023.eqiad.wmnet with OS trixie
* 01:52 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 01:30 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 01:15 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1023.eqiad.wmnet with reason: host reimage
* 01:11 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1023.eqiad.wmnet with reason: host reimage
* 00:59 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1023.eqiad.wmnet with OS trixie
* 00:49 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1023.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:37 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1023.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:37 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1023
* 00:36 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1023
* 00:27 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 00:27 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1023] - vriley@cumin1003&quot;
* 00:27 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1023] - vriley@cumin1003&quot;
* 00:20 vriley@cumin1003: START - Cookbook sre.dns.netbox

== 2026-05-07 ==
* 23:25 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1022.eqiad.wmnet with OS trixie
* 23:25 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 23:24 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 23:09 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1022.eqiad.wmnet with reason: host reimage
* 23:05 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1022.eqiad.wmnet with reason: host reimage
* 22:53 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1022.eqiad.wmnet with OS trixie
* 22:25 amastilovic@deploy1003: Finished deploy [analytics/refinery@b38efb1] (thin): Regular analytics weekly train THIN [analytics/refinery@b38efb19] (duration: 01m 53s)
* 22:23 amastilovic@deploy1003: Started deploy [analytics/refinery@b38efb1] (thin): Regular analytics weekly train THIN [analytics/refinery@b38efb19]
* 22:23 amastilovic@deploy1003: Finished deploy [analytics/refinery@b38efb1]: Regular analytics weekly train [analytics/refinery@b38efb19] (duration: 03m 52s)
* 22:19 amastilovic@deploy1003: Started deploy [analytics/refinery@b38efb1]: Regular analytics weekly train [analytics/refinery@b38efb19]
* 22:18 amastilovic@deploy1003: Finished deploy [analytics/refinery@b38efb1] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@b38efb19] (duration: 01m 55s)
* 22:16 amastilovic@deploy1003: Started deploy [analytics/refinery@b38efb1] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@b38efb19]
* {{safesubst:SAL entry|1=21:27 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284828{{!}}Upgrading webonyx/graphql-php (v15.31.5 =&gt; v15.32.3)]], [[gerrit:1284834{{!}}composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T425731)}}
* 21:23 cscott@deploy1003: cscott: Continuing with deployment
* 21:17 cscott@deploy1003: cscott: Backport for [[gerrit:1284828{{!}}Upgrading webonyx/graphql-php (v15.31.5 =&gt; v15.32.3)]], [[gerrit:1284834{{!}}composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T425731)]] synced to the t
* {{safesubst:SAL entry|1=21:16 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1284828{{!}}Upgrading webonyx/graphql-php (v15.31.5 =&gt; v15.32.3)]], [[gerrit:1284834{{!}}composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837{{!}}Bump wikimedia/parsoid to 0.24.0-a2 (T425731)]}}
* 20:53 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1021.eqiad.wmnet with OS trixie
* 20:53 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 20:49 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284575{{!}}Revert &quot;Enable mobile editor abandonment survey on enwiki&quot; (T424102)]], [[gerrit:1284702{{!}}Remove duplicate definition of EditCheckAction#isTagged (T425583)]], [[gerrit:1284703{{!}}Save action filtering info in ContentBranchNodeCheck#onDocumentChange (T425583)]] (duration: 06m 38s)
* 20:48 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003&quot;
* 20:45 kemayo@deploy1003: esanders, kemayo: Continuing with deployment
* 20:44 kemayo@deploy1003: esanders, kemayo: Backport for [[gerrit:1284575{{!}}Revert &quot;Enable mobile editor abandonment survey on enwiki&quot; (T424102)]], [[gerrit:1284702{{!}}Remove duplicate definition of EditCheckAction#isTagged (T425583)]], [[gerrit:1284703{{!}}Save action filtering info in ContentBranchNodeCheck#onDocumentChange (T425583)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v
* 20:42 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1284575{{!}}Revert &quot;Enable mobile editor abandonment survey on enwiki&quot; (T424102)]], [[gerrit:1284702{{!}}Remove duplicate definition of EditCheckAction#isTagged (T425583)]], [[gerrit:1284703{{!}}Save action filtering info in ContentBranchNodeCheck#onDocumentChange (T425583)]]
* 20:41 Krinkle: krinkle@deploy1003$ mwscript deleteEqualMessages.php commonswiki
* 20:41 Krinkle: krinkle@deploy1003$ mwscript deleteEqualMessages.php nlwiki
* 20:34 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1021.eqiad.wmnet with reason: host reimage
* 20:30 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1021.eqiad.wmnet with reason: host reimage
* 20:29 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:28 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:14 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284687{{!}}Provide page context for LintErrorChecker (T419596)]], [[gerrit:1284771{{!}}Make email confirmation banner a standalone RL module (T425677)]] (duration: 07m 18s)
* 20:10 arlolra@deploy1003: arlolra, mmartorana: Continuing with deployment
* 20:10 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1021.eqiad.wmnet with OS trixie
* 20:09 arlolra@deploy1003: arlolra, mmartorana: Backport for [[gerrit:1284687{{!}}Provide page context for LintErrorChecker (T419596)]], [[gerrit:1284771{{!}}Make email confirmation banner a standalone RL module (T425677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:07 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1284687{{!}}Provide page context for LintErrorChecker (T419596)]], [[gerrit:1284771{{!}}Make email confirmation banner a standalone RL module (T425677)]]
* 20:02 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1022.eqiad.wmnet with OS trixie
* 19:59 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:57 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:09 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1022.eqiad.wmnet with OS trixie
* 19:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1022.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1022.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:51 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1022
* 18:49 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1022
* 18:49 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:49 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1022~] - vriley@cumin1003&quot;
* 18:49 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1022~] - vriley@cumin1003&quot;
* 18:45 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 18:26 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 18:26 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 18:25 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 18:24 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 18:22 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 18:22 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 18:21 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 18:21 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 18:20 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 18:19 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 18:19 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 18:18 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 18:17 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 18:06 cdanis@dns1005: END - running authdns-update
* 18:04 cdanis@dns1005: START - running authdns-update
* 18:02 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1237662{{!}}Profiler: Set explicit &quot;excimer-wall&quot; redis channel instead of concat]] (duration: 29m 24s)
* 18:02 brennen: 1.47.0-wmf.1 train status ([[phab:T423910|T423910]]): blockers resolved, rolling to all wikis
* 17:59 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 17:58 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 17:51 krinkle@deploy1003: krinkle: Continuing with deployment
* 17:50 krinkle@deploy1003: krinkle: Backport for [[gerrit:1237662{{!}}Profiler: Set explicit &quot;excimer-wall&quot; redis channel instead of concat]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:45 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 17:45 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 17:33 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1237662{{!}}Profiler: Set explicit &quot;excimer-wall&quot; redis channel instead of concat]]
* 17:32 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 17:32 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 17:06 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet,db1216.eqiad.wmnet with reason: restart
* 16:50 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2199.codfw.wmnet,db1245.eqiad.wmnet with reason: restart
* 16:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 16:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 16:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 16:35 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 16:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 16:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 16:32 jynus: restarting backup1-* database primary hosts
* 16:30 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2183.codfw.wmnet,db1204.eqiad.wmnet with reason: restart
* 16:25 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 36 hosts with reason: restart
* 16:14 sukhe@dns1004: END - running authdns-update
* 16:13 sukhe@dns1004: START - running authdns-update
* 16:13 sukhe@dns1004: START - running authdns-update
* 16:12 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 16:02 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and A:ulsfo and (A:dnsbox)
* 16:01 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 15:50 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on ms-backup[2003-2004].codfw.wmnet,ms-backup[1003-1004].eqiad.wmnet with reason: restart
* 15:44 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and A:ulsfo and (A:dnsbox)
* 15:32 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 15:32 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 15:31 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:31 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 15:31 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:31 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 15:24 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:24 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009.ulsfo.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 39 hosts
* 15:22 sukhe@cumin1003: START - Cookbook sre.hosts.remove-downtime for 39 hosts
* 15:18 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:18 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:15 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4050.ulsfo.wmnet
* 15:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:12 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified]
* 15:06 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 15:06 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 15:06 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 15:05 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 15:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:03 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 15:03 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 15:03 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:01 akhatun: Deployed refinery using scap, then deployed onto hdfs
* 14:58 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
* 14:54 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 14:54 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 14:54 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 14:54 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 14:53 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 14:53 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 14:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:52 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 14:52 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 14:50 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 14:44 akhatun@deploy1003: Finished deploy [analytics/refinery@4734c67] (thin): Regular analytics weekly train THIN [analytics/refinery@4734c67c] (duration: 02m 01s)
* 14:43 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 14:43 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 14:42 akhatun@deploy1003: Started deploy [analytics/refinery@4734c67] (thin): Regular analytics weekly train THIN [analytics/refinery@4734c67c]
* 14:40 akhatun@deploy1003: Finished deploy [analytics/refinery@4734c67]: Regular analytics weekly train [analytics/refinery@4734c67c] (duration: 04m 38s)
* 14:40 jasmine@cumin2002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
* 14:37 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 14:36 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 14:36 akhatun@deploy1003: Started deploy [analytics/refinery@4734c67]: Regular analytics weekly train [analytics/refinery@4734c67c]
* 14:35 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 14:35 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 14:33 akhatun@deploy1003: Finished deploy [analytics/refinery@4734c67] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4734c67c] (duration: 01m 54s)
* 14:32 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: cluster=dnsbox,dc=ulsfo [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 14:32 slyngshede@dns1004: END - running authdns-update
* 14:32 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:31 akhatun@deploy1003: Started deploy [analytics/refinery@4734c67] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4734c67c]
* 14:31 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:31 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 14:30 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 14:30 slyngshede@dns1004: START - running authdns-update
* 14:30 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 14:30 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 14:30 akhatun: Deploying Refinery at {{Gerrit|4734c67}} for weekly deployment train
* 14:30 jmm@dns1004: END - running authdns-update
* 14:29 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 14:28 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 14:28 jmm@dns1004: START - running authdns-update
* 14:28 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:28 slyngshede@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating DNS snippets - slyngshede@cumin1003&quot;
* 14:28 slyngshede@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: updating DNS snippets - slyngshede@cumin1003&quot;
* 14:26 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 14:26 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 14:25 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 14:25 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 14:24 slyngshede@cumin1003: START - Cookbook sre.dns.netbox
* 14:12 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
* 14:12 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 14:12 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 14:10 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 14:10 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 13:53 jasmine@cumin2002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
* 13:34 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284553{{!}}Enable staggered rollout for IRS on enwiki (T424008)]], [[gerrit:1284569{{!}}Fix when user is considered exposed to the feature in the experiment (T424075)]] (duration: 09m 05s)
* 13:30 stran@deploy1003: stran: Continuing with deployment
* 13:27 stran@deploy1003: stran: Backport for [[gerrit:1284553{{!}}Enable staggered rollout for IRS on enwiki (T424008)]], [[gerrit:1284569{{!}}Fix when user is considered exposed to the feature in the experiment (T424075)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1284553{{!}}Enable staggered rollout for IRS on enwiki (T424008)]], [[gerrit:1284569{{!}}Fix when user is considered exposed to the feature in the experiment (T424075)]]
* 13:23 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 13:10 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284547{{!}}Remove the progress bar]], [[gerrit:1275467{{!}}mc: Set server, instead of host and port, for wgWikiLambdaObjectCaches (T423311)]] (duration: 06m 55s)
* 13:06 jforrester@deploy1003: rzl, jforrester, hartman: Continuing with deployment
* 13:05 jforrester@deploy1003: rzl, jforrester, hartman: Backport for [[gerrit:1284547{{!}}Remove the progress bar]], [[gerrit:1275467{{!}}mc: Set server, instead of host and port, for wgWikiLambdaObjectCaches (T423311)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:03 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1284547{{!}}Remove the progress bar]], [[gerrit:1275467{{!}}mc: Set server, instead of host and port, for wgWikiLambdaObjectCaches (T423311)]]
* 13:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=dns4004.wikimedia.org [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 12:58 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:55 sukhe@cumin1003: START - Cookbook sre.dns.netbox
* 12:51 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 12:51 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 12:51 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 12:50 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 12:45 sukhe@dns1004: FAIL - running authdns-update
* 12:44 sukhe@dns1004: START - running authdns-update
* 12:30 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1205.eqiad.wmnet with OS trixie
* 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host install5004.wikimedia.org
* 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host install5004.wikimedia.org with OS bookworm
* 12:23 slyngshede@dns1004: FAIL - running authdns-update
* 12:21 slyngshede@dns1004: START - running authdns-update
* 12:18 moritzm: installing init-system-helpers bugfix updates from Bookworm point release
* 12:18 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;add lswtest back as being planned won't work - cmooney@cumin1003&quot;
* 12:17 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;add lswtest back as being planned won't work - cmooney@cumin1003&quot;
* 12:12 slyngshede@dns1004: FAIL - running authdns-update
* 12:11 slyngshede@dns1004: START - running authdns-update
* 12:11 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 12:11 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 12:11 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: cluster=dnsbox,dc=ulsfo,service=authdns-update [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 12:08 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1205.eqiad.wmnet with reason: host reimage
* 12:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: after reimage to trixie
* 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on install5004.wikimedia.org with reason: host reimage
* 12:02 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 12:02 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 12:02 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1205.eqiad.wmnet with reason: host reimage
* 12:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on install5004.wikimedia.org with reason: host reimage
* 11:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1227: after reimage to trixie
* 11:47 root@cumin1003: START - Cookbook sre.hosts.reimage for host db1205.eqiad.wmnet with OS trixie
* 11:46 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1205.eqiad.wmnet with reason: reimage
* 11:43 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 11:43 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 11:40 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2184.codfw.wmnet with OS trixie
* 11:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
* 11:36 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 11:35 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 11:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
* 11:20 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2168: after reimage to trixie
* 11:19 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2184.codfw.wmnet with reason: host reimage
* 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2168.codfw.wmnet with OS trixie
* 11:16 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 11:15 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 11:15 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2184.codfw.wmnet with reason: host reimage
* 11:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92412 and previous config saved to /var/cache/conftool/dbconfig/20260507-111424-fceratto.json
* 11:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1227: after reimage to trixie
* 11:11 moritzm: instaling modsecurity-apache security updates
* 11:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1227.eqiad.wmnet with OS trixie
* 11:07 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host install5004.wikimedia.org with OS bookworm
* 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92409 and previous config saved to /var/cache/conftool/dbconfig/20260507-110415-fceratto.json
* 11:04 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 11:04 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 11:03 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 11:03 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:59 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 10:59 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:59 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 10:58 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db2184
* 10:58 root@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2184
* 10:57 root@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db2184
* 10:57 root@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2184.codfw.wmnet 129.32.192.10.in-addr.arpa 9.2.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:57 root@cumin1003: START - Cookbook sre.dns.wipe-cache db2184.codfw.wmnet 129.32.192.10.in-addr.arpa 9.2.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:57 root@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:57 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 10:57 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 10:57 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 10:57 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:57 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:56 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284592{{!}}Close Russian Wikinews (T421796)]] (duration: 08m 40s)
* 10:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2168.codfw.wmnet with reason: host reimage
* 10:54 root@cumin1003: START - Cookbook sre.dns.netbox
* 10:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92407 and previous config saved to /var/cache/conftool/dbconfig/20260507-105407-fceratto.json
* 10:51 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 10:51 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2168.codfw.wmnet with reason: host reimage
* 10:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1284592{{!}}Close Russian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:49 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:49 root@cumin1003: START - Cookbook sre.hosts.move-vlan for host db2184
* 10:48 root@cumin1003: START - Cookbook sre.hosts.reimage for host db2184.codfw.wmnet with OS trixie
* 10:48 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1227.eqiad.wmnet with reason: host reimage
* 10:48 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 10:47 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 10:47 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1284592{{!}}Close Russian Wikinews (T421796)]]
* 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92406 and previous config saved to /var/cache/conftool/dbconfig/20260507-104359-fceratto.json
* 10:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1227.eqiad.wmnet with reason: host reimage
* 10:40 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2184.codfw.wmnet with reason: reimage
* 10:40 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 10:40 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 10:39 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 10:39 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:39 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 10:35 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:35 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92405 and previous config saved to /var/cache/conftool/dbconfig/20260507-103349-fceratto.json
* 10:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance
* 10:32 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2168.codfw.wmnet with OS trixie
* 10:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts hcaptcha-proxy5002.wikimedia.org
* 10:32 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:31 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: Reimage to Trixie
* 10:30 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2168: Reimage to Trixie
* 10:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2168.codfw.wmnet with reason: Reimage to Trixie
* 10:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2182: after reimage to trixie
* 10:28 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1227.eqiad.wmnet with OS trixie
* 10:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1227: Reimage to Trixie
* 10:26 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1227: Reimage to Trixie
* 10:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1227.eqiad.wmnet with reason: Reimage to Trixie
* 10:26 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1202: after reimage to trixie
* 10:21 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:21 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 10:20 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 10:16 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts hcaptcha-proxy5002.wikimedia.org
* 10:14 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 10:14 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 10:13 moritzm: rebalance ganti cluster in ulsfo following host reimages [[phab:T424686|T424686]]
* 10:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts hcaptcha-proxy5001.wikimedia.org
* 10:11 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:11 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 10:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast4006.wikimedia.org with OS trixie
* 10:10 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 10:04 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 10:03 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: hcaptcha-proxy5001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 10:03 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 09:59 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 09:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts hcaptcha-proxy5001.wikimedia.org
* 09:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast4006.wikimedia.org with reason: host reimage
* 09:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast4006.wikimedia.org with reason: host reimage
* 09:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2182: after reimage to trixie
* 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of hcaptcha-proxy4003.wikimedia.org to drbd
* 09:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2182.codfw.wmnet with OS trixie
* 09:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1202: after reimage to trixie
* 09:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1202.eqiad.wmnet with OS trixie
* 09:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
* 09:32 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of hcaptcha-proxy4003.wikimedia.org to drbd
* 09:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus4003.ulsfo.wmnet to drbd
* 09:25 elukey@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts sretest1006.eqiad.wmnet
* 09:24 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1006.eqiad.wmnet
* 09:19 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast4006.wikimedia.org with OS trixie
* 09:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
* 09:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM bast4006.wikimedia.org
* 09:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1202.eqiad.wmnet with reason: host reimage
* 09:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
* 09:11 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM bast4006.wikimedia.org
* 09:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2208: After reimage
* 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1202.eqiad.wmnet with reason: host reimage
* 08:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2182.codfw.wmnet with OS trixie
* 08:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1202.eqiad.wmnet with OS trixie
* 08:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1202: Reimage to Trixie
* 08:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2182: Reimage to Trixie
* 08:51 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2182: Reimage to Trixie
* 08:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2182.codfw.wmnet with reason: Reimage to Trixie
* 08:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1202: Reimage to Trixie
* 08:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1202.eqiad.wmnet with reason: Reimage to Trixie
* 08:42 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2144.codfw.wmnet
* 08:42 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:42 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2144.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 08:42 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2144.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 08:37 marostegui@cumin1003: START - Cookbook sre.dns.netbox
* 08:32 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2144.codfw.wmnet
* 08:29 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus4003.ulsfo.wmnet to drbd
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir4004.ulsfo.wmnet to drbd
* 08:28 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db2144 [[phab:T425522|T425522]]', diff saved to https://phabricator.wikimedia.org/P92389 and previous config saved to /var/cache/conftool/dbconfig/20260507-082822-marostegui.json
* 08:23 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2208: After reimage
* 08:23 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2208: After reimage
* 08:23 XioNoX: drmrs remove old v6 gateway IP
* 08:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: drmrs v6 gateway IPs change - ayounsi@cumin1003&quot;
* 08:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2208: After reimage
* 08:21 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: drmrs v6 gateway IPs change - ayounsi@cumin1003&quot;
* 08:17 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir4004.ulsfo.wmnet to drbd
* 08:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 08:12 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync
* 08:12 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync
* 08:12 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 08:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
* 08:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
* 08:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 08:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti4008.ulsfo.wmnet to cluster ulsfo02 and group 01
* 07:54 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269465{{!}}search: add alt. completion indices to test keyword tokenizer (2/2) (T420427)]] (duration: 09m 46s)
* 07:49 dcausse@deploy1003: dcausse: Continuing with deployment
* 07:46 dcausse@deploy1003: dcausse: Backport for [[gerrit:1269465{{!}}search: add alt. completion indices to test keyword tokenizer (2/2) (T420427)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:44 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow4003.ulsfo.wmnet to drbd
* 07:44 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1269465{{!}}search: add alt. completion indices to test keyword tokenizer (2/2) (T420427)]]
* 07:32 moritzm: installing apache2 security updates
* 07:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow4003.ulsfo.wmnet to drbd
* 07:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM testvm2005.codfw.wmnet
* 07:23 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM testvm2005.codfw.wmnet
* 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:46 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:46 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir4003.ulsfo.wmnet to drbd
* 06:42 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4006.ulsfo.wmnet to cluster ulsfo02 and group 01
* 06:41 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti4006.ulsfo.wmnet to cluster ulsfo02 and group 01
* 06:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: after reimage to trixie
* 05:54 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2207: after reimage to trixie
* 05:51 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2207.codfw.wmnet with OS trixie
* 05:33 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2208.codfw.wmnet with OS trixie
* 05:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2207.codfw.wmnet with reason: host reimage
* 05:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2207.codfw.wmnet with reason: host reimage
* 05:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2208.codfw.wmnet with reason: host reimage
* 05:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2208.codfw.wmnet with reason: host reimage
* 05:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2207.codfw.wmnet with OS trixie
* 05:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: Reimage to Trixie
* 05:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2207: Reimage to Trixie
* 05:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2207.codfw.wmnet with reason: Reimage to Trixie
* 04:52 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2207 [[phab:T424848|T424848]]', diff saved to https://phabricator.wikimedia.org/P92383 and previous config saved to /var/cache/conftool/dbconfig/20260507-045219-marostegui.json
* 04:51 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2204 to s2 primary [[phab:T424848|T424848]]', diff saved to https://phabricator.wikimedia.org/P92382 and previous config saved to /var/cache/conftool/dbconfig/20260507-045141-marostegui.json
* 04:51 marostegui: Starting s2 codfw failover from db2207 to db2204 - [[phab:T424848|T424848]]
* 04:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T424848|T424848]]
* 04:46 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2204 with weight 0 [[phab:T424848|T424848]]', diff saved to https://phabricator.wikimedia.org/P92381 and previous config saved to /var/cache/conftool/dbconfig/20260507-044651-marostegui.json
* 04:46 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2208.codfw.wmnet with OS trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s)
* 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:15 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281485{{!}}Drop some unneeded wikinews configs (T421796)]] (duration: 12m 57s)
* 01:09 zabe@deploy1003: zabe: Continuing with deployment
* 01:09 zabe@deploy1003: zabe: Backport for [[gerrit:1281485{{!}}Drop some unneeded wikinews configs (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 01:02 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281485{{!}}Drop some unneeded wikinews configs (T421796)]]
* 01:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1021.eqiad.wmnet with OS trixie
* 00:43 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277783{{!}}Undeploy GoogleNewsSitemap (T421798)]] (duration: 33m 54s)
* 00:31 zabe@deploy1003: zabe: Continuing with deployment
* 00:29 zabe@deploy1003: zabe: Backport for [[gerrit:1277783{{!}}Undeploy GoogleNewsSitemap (T421798)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:10 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1277783{{!}}Undeploy GoogleNewsSitemap (T421798)]]

== 2026-05-06 ==
* 23:41 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1021.eqiad.wmnet with OS trixie
* 23:38 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1021.eqiad.wmnet with OS trixie
* 23:14 ladsgroup@deploy1003: Synchronized portals: Sync portals for removal of Wikinews (duration: 02m 22s)
* 23:12 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Sync portals for removal of Wikinews (duration: 06m 12s)
* 22:50 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284004{{!}}Close Spanish Wikinews (T421796)]] (duration: 07m 08s)
* 22:46 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:45 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1284004{{!}}Close Spanish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:43 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1284004{{!}}Close Spanish Wikinews (T421796)]]
* 22:33 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283872{{!}}Close English Wikinews (T421796)]] (duration: 06m 40s)
* 22:28 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:28 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283872{{!}}Close English Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283872{{!}}Close English Wikinews (T421796)]]
* 22:18 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host pc1021.eqiad.wmnet with OS trixie
* 22:14 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:14 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283972{{!}}UBN fix: guard entry.serverTiming before forEach (T425591)]] (duration: 06m 25s)
* 22:11 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:11 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:10 cjming@deploy1003: cjming: Continuing with deployment
* 22:10 cjming@deploy1003: cjming: Backport for [[gerrit:1283972{{!}}UBN fix: guard entry.serverTiming before forEach (T425591)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:09 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:08 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:08 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1283972{{!}}UBN fix: guard entry.serverTiming before forEach (T425591)]]
* 22:06 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 22:05 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1021
* 22:04 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1021
* 21:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283953{{!}}Disable GNSM on dewikinews (T421798)]] (duration: 06m 56s)
* 21:48 zabe@deploy1003: zabe: Continuing with deployment
* 21:47 zabe@deploy1003: zabe: Backport for [[gerrit:1283953{{!}}Disable GNSM on dewikinews (T421798)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:45 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1283953{{!}}Disable GNSM on dewikinews (T421798)]]
* 21:31 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:29 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:28 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:27 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1012.eqiad.wmnet with OS trixie
* 21:26 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:24 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:22 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:17 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:15 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:14 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:12 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:11 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:11 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1021] - vriley@cumin1003&quot;
* 21:11 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  [pc1021] - vriley@cumin1003&quot;
* 21:07 vriley@cumin1003: START - Cookbook sre.dns.netbox
* 21:06 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc1021
* 21:05 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host pc1021
* 21:04 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 20:29 vriley@cumin1003: START - Cookbook sre.hosts.provision for host pc1021.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281526{{!}}Replace use of $wgRequest (T336703)]] (duration: 09m 12s)
* 20:25 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 20:24 catrope@deploy1003: catrope, somerandomdeveloper: Continuing with deployment
* 20:21 catrope@deploy1003: catrope, somerandomdeveloper: Backport for [[gerrit:1281526{{!}}Replace use of $wgRequest (T336703)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:19 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1281526{{!}}Replace use of $wgRequest (T336703)]]
* 20:14 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie
* 20:00 bking@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1012.eqiad.wmnet with OS trixie
* 19:57 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie
* 19:30 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS trixie
* 19:24 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1012.eqiad.wmnet with OS trixie
* 19:23 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4050.ulsfo.wmnet with OS trixie
* 19:14 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS bookworm
* 19:05 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 19:01 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage
* 18:59 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 18:59 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 18:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 18:55 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 18:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 18:55 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 18:54 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 18:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 18:54 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 18:53 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 18:53 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:49 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie
* 18:48 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 18:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 18:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:42 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 18:42 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 18:42 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 18:41 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 18:40 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 18:40 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 18:39 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:37 dzahn@dns1005: END - running authdns-update
* 18:35 dzahn@dns1005: START - running authdns-update
* 18:33 brennen: 1.47.0-wmf.1 train status ([[phab:T423910|T423910]]): blockers resolved, rolling to group1
* 18:31 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp4050.ulsfo.wmnet with OS trixie
* 18:29 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS bookworm
* 18:02 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-23-ulsfo
* 18:01 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-23-ulsfo
* 17:59 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: restart to test fixes from [[phab:T425301|T425301]] - bking@cumin2002
* 17:55 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device asw1-23-ulsfo
* 17:55 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-23-ulsfo
* 17:37 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 17:32 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 17:31 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 17:28 topranks: rebooting asw1-23-ulsfo to upgrade SR-Linux OS on switch [[phab:T408892|T408892]]
* 17:27 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw1-23-ulsfo,asw1-23-ulsfo IPv6 with reason: upgrading sr-linux on asw1-23-ulsfo
* 17:20 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 17:18 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
* 17:18 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
* 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
* 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
* 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
* 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
* 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
* 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
* 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 17:06 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 17:02 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on 39 hosts with reason: ulsfo depooled for switch work
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw1-22-ulsfo,asw1-22-ulsfo IPv6 with reason: upgrading sr-linux on asw1-23-ulsfo
* 16:52 topranks: rebooting asw1-22-ulsfo to upgrade SR-Linux OS on switch [[phab:T408892|T408892]]
* 16:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:40 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS trixie
* 16:39 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:37 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS bookworm
* 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:28 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS bookworm
* 16:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:28 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 16:09 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 16:04 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4004.wikimedia.org with reason: host reimage
* 15:58 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 15:57 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns4004.wikimedia.org with reason: host reimage
* 15:38 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS bookworm
* 15:35 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS bookworm
* 15:30 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=0) Change Confluent distribution for Kafka A:kafka-main-eqiad cluster: Change Confluent distribution.
* 15:08 sukhe: sudo cumin -b1 -s5 &quot;C:bird and not dns4004*&quot; &quot;run-puppet-agent --enable 'merging CR 1282958'&quot;
* 15:08 jasmine@cumin2002: START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-main-eqiad cluster: Change Confluent distribution.
* 15:06 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283805{{!}}Close Chinese Wikinews (T421796)]] (duration: 06m 41s)
* 15:02 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 15:02 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 15:01 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:01 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283805{{!}}Close Chinese Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:59 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283805{{!}}Close Chinese Wikinews (T421796)]]
* 14:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts durum5002.eqsin.wmnet
* 14:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:53 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS trixie
* 14:51 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:45 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 14:36 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:36 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:35 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.wikimedia.org [reason: testing bird change]
* 14:31 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.wikimedia.org [reason: testing bird change]
* 14:30 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283050{{!}}Add user_groups to editAttemptStep schema (T424010)]] (duration: 11m 16s)
* 14:28 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
* 14:26 kharlan@deploy1003: kharlan: Continuing with deployment
* 14:25 sukhe: sudo cumin &quot;C:bird&quot; &quot;disable-puppet 'merging CR 1282958'&quot;
* 14:23 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
* 14:22 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:21 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:21 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1283050{{!}}Add user_groups to editAttemptStep schema (T424010)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:21 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts durum5002.eqsin.wmnet
* 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4008.ulsfo.wmnet with OS bookworm
* 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 14:20 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 14:20 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:19 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1283050{{!}}Add user_groups to editAttemptStep schema (T424010)]]
* 14:19 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:18 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts durum5001.eqsin.wmnet
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:15 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283783{{!}}Close German Wikinews (T421796)]] (duration: 06m 40s)
* 14:13 dmartin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:13 dmartin@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:12 dmartin@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:12 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 14:11 dmartin@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4046.ulsfo.wmnet with OS trixie
* 14:10 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 14:10 dmartin@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:10 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283783{{!}}Close German Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:10 dmartin@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:09 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:08 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:08 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283783{{!}}Close German Wikinews (T421796)]]
* 14:08 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 14:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283751{{!}}Close French Wikinews (T421796)]] (duration: 11m 28s)
* 14:02 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts durum5001.eqsin.wmnet
* 14:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage
* 13:56 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 13:56 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage
* 13:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283751{{!}}Close French Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:55 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS trixie
* 13:53 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: restart to test fixes from [[phab:T425301|T425301]] - bking@cumin2002
* 13:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1194: after reimage to trixie
* 13:51 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283751{{!}}Close French Wikinews (T421796)]]
* 13:45 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4046.ulsfo.wmnet with reason: host reimage
* 13:45 jgreen@dns1004: END - running authdns-update
* 13:44 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283028{{!}}Add messages related to mandatory 2FA for more groups (T423119)]] (duration: 30m 53s)
* 13:44 jgreen@dns1004: START - running authdns-update
* 13:41 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4046.ulsfo.wmnet with reason: host reimage
* 13:39 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 13:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti4008.ulsfo.wmnet with OS bookworm
* 13:35 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) ganeti4008.ulsfo.wmnet on all recursors
* 13:34 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti4008.ulsfo.wmnet on all recursors
* 13:32 alexsanford@deploy1003: alexsanford: Continuing with deployment
* 13:32 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:31 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 13:31 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1283028{{!}}Add messages related to mandatory 2FA for more groups (T423119)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:28 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:28 jmm@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4008.ulsfo.wmnet']
* 13:27 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:26 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 13:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 13:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:20 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:19 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) ganeti4008.mgmt.ulsfo.wmnet on all recursors
* 13:19 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS trixie
* 13:19 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti4008.mgmt.ulsfo.wmnet on all recursors
* 13:19 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:18 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for ganeti4008 mgmt - cmooney@cumin1003&quot;
* 13:18 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add entry for ganeti4008 mgmt - cmooney@cumin1003&quot;
* 13:15 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 13:14 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 13:14 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 13:13 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1283028{{!}}Add messages related to mandatory 2FA for more groups (T423119)]]
* 13:12 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS trixie
* 13:11 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:08 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
* 13:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1194: after reimage to trixie
* 13:05 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 13:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1194.eqiad.wmnet with OS trixie
* 12:49 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS trixie
* 12:45 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 12:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2012.codfw.wmnet with OS trixie
* 12:39 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 12:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
* 12:36 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: update
* 12:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
* 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2012.codfw.wmnet with reason: host reimage
* 12:21 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2012.codfw.wmnet with reason: host reimage
* 12:20 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1194.eqiad.wmnet with OS trixie
* 12:20 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4048.ulsfo.wmnet with OS trixie
* 12:16 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4050.ulsfo.wmnet with OS trixie
* 12:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 12:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 12:14 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2011.codfw.wmnet with OS trixie
* 12:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283735{{!}}Close Polish Wikinews (T421796)]] (duration: 06m 28s)
* 12:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 12:07 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 12:07 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283735{{!}}Close Polish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:05 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb2012.codfw.wmnet with OS trixie
* 12:05 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283735{{!}}Close Polish Wikinews (T421796)]]
* 12:05 jclark@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 11:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2011.codfw.wmnet with reason: host reimage
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet
* 11:53 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4048.ulsfo.wmnet with reason: host reimage
* 11:50 moritzm: installing openjdk-17 security updates
* 11:50 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 11:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92374 and previous config saved to /var/cache/conftool/dbconfig/20260506-114919-fceratto.json
* 11:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet
* 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1194: Reimage to Trixie
* 11:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2160.codfw.wmnet with reason: Reboot
* 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1194: Reimage to Trixie
* 11:44 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2011.codfw.wmnet with reason: host reimage
* 11:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1194.eqiad.wmnet with reason: Reimage to Trixie
* 11:42 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4048.ulsfo.wmnet with reason: host reimage
* 11:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4006.ulsfo.wmnet with OS bookworm
* 11:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 11:41 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4050.ulsfo.wmnet with reason: host reimage
* 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92372 and previous config saved to /var/cache/conftool/dbconfig/20260506-113910-fceratto.json
* 11:30 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb2011.codfw.wmnet with OS trixie
* 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92371 and previous config saved to /var/cache/conftool/dbconfig/20260506-112903-fceratto.json
* 11:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002&quot;
* 11:20 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4050.ulsfo.wmnet with OS trixie
* 11:19 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4048.ulsfo.wmnet with OS trixie
* 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92370 and previous config saved to /var/cache/conftool/dbconfig/20260506-111854-fceratto.json
* 11:14 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS trixie
* 11:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4042.ulsfo.wmnet with OS trixie
* 11:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: Reboot
* 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4006.ulsfo.wmnet with reason: host reimage
* 10:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4006.ulsfo.wmnet with reason: host reimage
* 10:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage
* 10:44 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4042.ulsfo.wmnet with reason: host reimage
* 10:40 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage
* 10:39 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4042.ulsfo.wmnet with reason: host reimage
* 10:33 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti4006.ulsfo.wmnet with OS bookworm
* 10:29 jmm@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:23 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:22 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4006.ulsfo.wmnet']
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92369 and previous config saved to /var/cache/conftool/dbconfig/20260506-101836-fceratto.json
* 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92368 and previous config saved to /var/cache/conftool/dbconfig/20260506-101808-fceratto.json
* 10:17 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS trixie
* 10:16 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4042.ulsfo.wmnet with OS trixie
* 10:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4040.ulsfo.wmnet with OS trixie
* 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92367 and previous config saved to /var/cache/conftool/dbconfig/20260506-100800-fceratto.json
* 09:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92366 and previous config saved to /var/cache/conftool/dbconfig/20260506-095752-fceratto.json
* 09:55 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92365 and previous config saved to /var/cache/conftool/dbconfig/20260506-094744-fceratto.json
* 09:45 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4040.ulsfo.wmnet with reason: host reimage
* 09:40 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4040.ulsfo.wmnet with reason: host reimage
* 09:38 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:32 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:31 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:29 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:29 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006/8 mgmt - ayounsi@cumin1003&quot;
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.provision for host ganeti4006.mgmt.ulsfo.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92364 and previous config saved to /var/cache/conftool/dbconfig/20260506-092414-fceratto.json
* 09:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance
* 09:23 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006/8 mgmt - ayounsi@cumin1003&quot;
* 09:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92363 and previous config saved to /var/cache/conftool/dbconfig/20260506-092345-fceratto.json
* 09:17 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 09:17 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp4040.ulsfo.wmnet with OS trixie
* 09:16 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
* 09:15 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup2005.codfw.wmnet with reason: update
* 09:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repool ms2 [[phab:T418979|T418979]]ç', diff saved to https://phabricator.wikimedia.org/P92362 and previous config saved to /var/cache/conftool/dbconfig/20260506-091513-marostegui.json
* 09:14 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 09:14 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2253: Replacing HW [[phab:T418979|T418979]]
* 09:14 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99)
* 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
* 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2253: Replacing HW [[phab:T418979|T418979]]
* 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039', diff saved to https://phabricator.wikimedia.org/P92361 and previous config saved to /var/cache/conftool/dbconfig/20260506-091337-fceratto.json
* 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039', diff saved to https://phabricator.wikimedia.org/P92360 and previous config saved to /var/cache/conftool/dbconfig/20260506-090329-fceratto.json
* 09:03 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281894{{!}}Correctly support new file tables in RevisionDeleteUser (T424553)]] (duration: 08m 44s)
* 08:59 zabe@deploy1003: zabe: Continuing with deployment
* 08:56 zabe@deploy1003: zabe: Backport for [[gerrit:1281894{{!}}Correctly support new file tables in RevisionDeleteUser (T424553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281894{{!}}Correctly support new file tables in RevisionDeleteUser (T424553)]]
* 08:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92359 and previous config saved to /var/cache/conftool/dbconfig/20260506-085321-fceratto.json
* 08:43 marostegui@cumin1003: dbctl commit (dc=all): 'Add db2253 to ms2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92358 and previous config saved to /var/cache/conftool/dbconfig/20260506-084337-marostegui.json
* 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92357 and previous config saved to /var/cache/conftool/dbconfig/20260506-083841-fceratto.json
* 08:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance
* 08:29 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2208.codfw.wmnet with OS trixie
* 08:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2208.codfw.wmnet with OS trixie
* 08:08 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2208.codfw.wmnet with OS trixie
* 08:06 awight: EU morning deployment is done
* 08:00 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2144.codfw.wmnet,db1151.eqiad.wmnet with reason: Replacing hw
* 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2144: Replacing HW [[phab:T418979|T418979]]
* 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 07:58 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
* 07:58 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2144: Replacing HW [[phab:T418979|T418979]]
* 07:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2208.codfw.wmnet with OS trixie
* 07:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283101{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]], [[gerrit:1283037{{!}}search: fix alt. completion indices to test keyword tokenizer (T420427)]], [[gerrit:1283041{{!}}search: enable Latin-to-Devanagari transliteration second-chance (T425018)]] (duration: 08m 58s)
* 07:36 awight@deploy1003: wmde-fisch, awight, dcausse: Continuing with deployment
* 07:33 awight@deploy1003: wmde-fisch, awight, dcausse: Backport for [[gerrit:1283101{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]], [[gerrit:1283037{{!}}search: fix alt. completion indices to test keyword tokenizer (T420427)]], [[gerrit:1283041{{!}}search: enable Latin-to-Devanagari transliteration second-chance (T425018)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can
* 07:31 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1283101{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]], [[gerrit:1283037{{!}}search: fix alt. completion indices to test keyword tokenizer (T420427)]], [[gerrit:1283041{{!}}search: enable Latin-to-Devanagari transliteration second-chance (T425018)]]
* 07:26 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283033{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]] (duration: 07m 37s)
* 07:22 awight@deploy1003: awight, lilients: Continuing with deployment
* 07:21 awight@deploy1003: awight, lilients: Backport for [[gerrit:1283033{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:19 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1283033{{!}}VE: Avoid counting all refs when listIndex is undefined (T425433)]]
* 07:14 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4008.ulsfo.wmnet
* 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4008.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4008.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 06:55 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 06:54 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1191: after reimage to trixie
* 06:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1189: after reimage to trixie
* 06:48 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti4008.ulsfo.wmnet
* 06:48 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4006.ulsfo.wmnet
* 06:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 06:30 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ganeti4006.ulsfo.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 06:26 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 06:20 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti4006.ulsfo.wmnet
* 05:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2208.codfw.wmnet with reason: Idrac issues [[phab:T425506|T425506]]
* 05:43 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
* 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
* 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
* 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
* 05:26 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2208: Reimage to Trixie
* 05:26 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2208: Reimage to Trixie
* 05:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2208.codfw.wmnet with reason: Reimage to Trixie
* 05:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2208: Reimage to Trixie
* 05:24 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2208: Reimage to Trixie
* 05:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2208.codfw.wmnet with reason: Reimage to Trixie
* 05:23 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1191.eqiad.wmnet with OS trixie
* 05:22 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1191: Reimage to Trixie
* 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1191: Reimage to Trixie
* 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Reimage to Trixie
* 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1189.eqiad.wmnet with OS trixie
* 05:14 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1189: Reimage to Trixie
* 05:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1189: Reimage to Trixie
* 05:14 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1189.eqiad.wmnet with reason: Reimage to Trixie
* 05:11 marostegui@dns1004: END - running authdns-update
* 05:09 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1189 [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92345 and previous config saved to /var/cache/conftool/dbconfig/20260506-050948-marostegui.json
* 05:09 marostegui@dns1004: START - running authdns-update
* 05:08 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1223 to s3 primary and set section read-write [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92344 and previous config saved to /var/cache/conftool/dbconfig/20260506-050816-marostegui.json
* 05:07 marostegui@cumin1003: dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92343 and previous config saved to /var/cache/conftool/dbconfig/20260506-050755-marostegui.json
* 05:06 marostegui: Starting s3 eqiad failover from db1189 to db1223 - [[phab:T425318|T425318]]
* 05:03 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 [[phab:T425318|T425318]]
* 05:03 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1223 with weight 0 [[phab:T425318|T425318]]', diff saved to https://phabricator.wikimedia.org/P92342 and previous config saved to /var/cache/conftool/dbconfig/20260506-050342-marostegui.json
* 03:28 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 03:27 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 03:27 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 03:26 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 37s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:05 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1001.eqiad.wmnet with OS trixie
* 00:54 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283061{{!}}Close Dutch Wikinews (T421796)]] (duration: 06m 26s)
* 00:49 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 00:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283061{{!}}Close Dutch Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283061{{!}}Close Dutch Wikinews (T421796)]]
* 00:45 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1001.eqiad.wmnet with reason: host reimage
* 00:41 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1001.eqiad.wmnet with reason: host reimage
* 00:27 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283117{{!}}Close Italian Wikinews (T421796)]] (duration: 07m 26s)
* 00:25 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1001
* 00:25 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1001
* 00:24 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1001.eqiad.wmnet with OS trixie
* 00:23 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 00:21 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283117{{!}}Close Italian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283117{{!}}Close Italian Wikinews (T421796)]]

== 2026-05-05 ==
* 23:31 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 23:30 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update ip addresses for nodes in rack 23 - pt1979@cumin2002&quot;
* 23:30 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update ip addresses for nodes in rack 23 - pt1979@cumin2002&quot;
* 23:26 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 22:54 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283104{{!}}Close Arabic Wikinews (T421796)]] (duration: 06m 58s)
* 22:50 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283104{{!}}Close Arabic Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283104{{!}}Close Arabic Wikinews (T421796)]]
* 22:43 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283103{{!}}Close Ukrainian Wikinews (T421796)]] (duration: 06m 28s)
* 22:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:39 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283103{{!}}Close Ukrainian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:37 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283103{{!}}Close Ukrainian Wikinews (T421796)]]
* 22:26 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283100{{!}}Close Romanian Wikinews (T421796)]] (duration: 07m 56s)
* 22:22 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283100{{!}}Close Romanian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283100{{!}}Close Romanian Wikinews (T421796)]]
* 22:16 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283099{{!}}Close Serbian Wikinews (T421796)]] (duration: 06m 45s)
* 22:12 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 22:11 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283099{{!}}Close Serbian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:09 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283099{{!}}Close Serbian Wikinews (T421796)]]
* 22:05 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283066{{!}}Close Persian Wikinews (T421796)]] (duration: 11m 07s)
* 21:59 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 21:58 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283066{{!}}Close Persian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:54 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283066{{!}}Close Persian Wikinews (T421796)]]
* 21:49 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281501{{!}}Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056{{!}}Legacy parser no longer varies by user thumbnail size. (T417513)]] (duration: 32m 55s)
* 21:36 arlolra@deploy1003: jdlrobson, mmartorana, arlolra: Continuing with deployment
* 21:33 arlolra@deploy1003: jdlrobson, mmartorana, arlolra: Backport for [[gerrit:1281501{{!}}Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056{{!}}Legacy parser no longer varies by user thumbnail size. (T417513)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:16 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1281501{{!}}Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056{{!}}Legacy parser no longer varies by user thumbnail size. (T417513)]]
* 20:59 dancy@deploy1003: Installation of scap version &quot;4.262.1&quot; completed for 2 hosts
* 20:57 dancy@deploy1003: Installing scap version &quot;4.262.1&quot; for 2 host(s)
* 20:57 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282930{{!}}hCaptcha: Add diagnostic context to script load error logs (T424496)]], [[gerrit:1282397{{!}}sectionCollapsing: Scroll to fragment target on init (T425290)]], [[gerrit:1282804{{!}}Errors added below ref list dirty when not responsive (T384599)]] (duration: 10m 59s)
* 20:52 arlolra@deploy1003: mpostoronca, h2o, awight, arlolra: Continuing with deployment
* 20:48 arlolra@deploy1003: mpostoronca, h2o, awight, arlolra: Backport for [[gerrit:1282930{{!}}hCaptcha: Add diagnostic context to script load error logs (T424496)]], [[gerrit:1282397{{!}}sectionCollapsing: Scroll to fragment target on init (T425290)]], [[gerrit:1282804{{!}}Errors added below ref list dirty when not responsive (T384599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve
* 20:46 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1282930{{!}}hCaptcha: Add diagnostic context to script load error logs (T424496)]], [[gerrit:1282397{{!}}sectionCollapsing: Scroll to fragment target on init (T425290)]], [[gerrit:1282804{{!}}Errors added below ref list dirty when not responsive (T384599)]]
* 20:41 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4038.ulsfo.wmnet with OS trixie
* 20:22 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283082{{!}}Enable WikiLove on shwiki (T424891)]], [[gerrit:1276814{{!}}Add wikibase.v1 module to the sandbox were it is present (T422403)]] (duration: 10m 30s)
* 20:20 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1002.eqiad.wmnet with OS trixie
* 20:18 arlolra@deploy1003: aaron, neriah, arlolra: Continuing with deployment
* 20:14 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4038.ulsfo.wmnet with reason: host reimage
* 20:13 arlolra@deploy1003: aaron, neriah, arlolra: Backport for [[gerrit:1283082{{!}}Enable WikiLove on shwiki (T424891)]], [[gerrit:1276814{{!}}Add wikibase.v1 module to the sandbox were it is present (T422403)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:12 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1283082{{!}}Enable WikiLove on shwiki (T424891)]], [[gerrit:1276814{{!}}Add wikibase.v1 module to the sandbox were it is present (T422403)]]
* 20:10 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:09 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:07 pt1979@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4038.ulsfo.wmnet with reason: host reimage
* 20:04 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: host reimage
* 19:57 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: host reimage
* 19:55 herron@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 19:55 herron@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 19:54 root@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
* 19:54 root@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
* 19:45 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 19:41 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1002
* 19:41 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging1002
* 19:41 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 19:41 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:40 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:39 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging1002
* 19:39 herron@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1002.eqiad.wmnet 142.32.64.10.in-addr.arpa 2.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 19:39 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging1002.eqiad.wmnet 142.32.64.10.in-addr.arpa 2.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 19:39 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:39 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1002 - herron@cumin1003&quot;
* 19:38 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1002 - herron@cumin1003&quot;
* 19:32 herron@cumin1003: START - Cookbook sre.dns.netbox
* 19:31 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 19:31 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:30 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:28 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1002
* 19:27 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1002.eqiad.wmnet with OS trixie
* 19:17 dancy@deploy1003: Installation of scap version &quot;4.262.0&quot; completed for 2 hosts
* 19:15 dancy@deploy1003: Installing scap version &quot;4.262.0&quot; for 2 host(s)
* 19:15 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: rebooting firewall in desperation
* 19:14 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 19:05 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;set correct vlan group in netbox for new ulsfo vlans - cmooney@cumin1003 - [[phab:T408892|T408892]]&quot;
* 19:05 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;set correct vlan group in netbox for new ulsfo vlans - cmooney@cumin1003 - [[phab:T408892|T408892]]&quot;
* 19:04 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie
* 19:03 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283063{{!}}Close Swedish Wikinews (T421796)]] (duration: 10m 59s)
* 18:56 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 18:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283063{{!}}Close Swedish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:52 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283063{{!}}Close Swedish Wikinews (T421796)]]
* 18:49 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 18:48 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:48 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:48 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:47 brennen@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.1  refs [[phab:T423910|T423910]] (duration: 36m 04s)
* 18:44 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie
* 18:44 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:44 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:44 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:44 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002&quot;
* 18:40 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:30 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 18:25 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie
* 18:14 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device pfw1a-codfw
* 18:13 cmooney@cumin1003: START - Cookbook sre.network.tls for network device pfw1a-codfw
* 18:13 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie
* 18:11 brennen@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.1  refs [[phab:T423910|T423910]]
* 18:10 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device pfw1a-eqiad
* 18:10 cmooney@cumin1003: START - Cookbook sre.network.tls for network device pfw1a-eqiad
* 18:06 brennen: 1.47.0-wmf.1 train status ([[phab:T423910|T423910]]): no current blockers, rolling to group0
* 18:04 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1003.eqiad.wmnet with OS trixie
* 17:44 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: host reimage
* 17:38 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: host reimage
* 17:33 herron@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 17:32 herron@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 17:23 root@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
* 17:23 root@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
* 17:21 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1003
* 17:21 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging1003
* 17:21 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging1003
* 17:20 herron@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:19 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:16 herron@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:15 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:15 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:15 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1003 - herron@cumin1003&quot;
* 17:15 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1003 - herron@cumin1003&quot;
* 17:12 herron@cumin1003: START - Cookbook sre.dns.netbox
* 17:09 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1003
* 17:08 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1003.eqiad.wmnet with OS trixie
* 17:05 sukhe: sudo cumin -b11 &quot;A:cp and not P&lt;nowiki&gt;{&lt;/nowiki&gt;cp2041* or cp2042*&lt;nowiki&gt;}&lt;/nowiki&gt; and not A:ulsfo&quot; &quot;run-puppet-agent --enable 'merging CR 1282979'&quot;
* 16:58 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049{{!}}Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]] (duration: 07m 25s)
* 16:53 sbassett@deploy1003: mstyles, sbassett: Continuing with deployment
* 16:52 sbassett@deploy1003: mstyles, sbassett: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049{{!}}Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdeb
* 16:50 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049{{!}}Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]]
* 16:38 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1283036{{!}}Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020{{!}}Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]]
* 16:19 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: sync
* 16:19 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: sync
* 16:19 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync
* 16:18 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: sync
* 16:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283040{{!}}Close Japanese Wikinews (T421796)]] (duration: 06m 16s)
* 16:07 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 16:07 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283040{{!}}Close Japanese Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:05 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283040{{!}}Close Japanese Wikinews (T421796)]]
* 16:01 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283039{{!}}Close Korean Wikinews (T421796)]] (duration: 07m 53s)
* 15:57 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:55 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: sync
* 15:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283039{{!}}Close Korean Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:55 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: sync
* 15:55 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync
* 15:54 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: sync
* 15:53 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283039{{!}}Close Korean Wikinews (T421796)]]
* 15:52 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283038{{!}}Close Finnish Wikinews (T421796)]] (duration: 06m 12s)
* 15:48 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:47 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283038{{!}}Close Finnish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:46 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283038{{!}}Close Finnish Wikinews (T421796)]]
* 15:42 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync
* 15:42 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync
* 15:39 dzahn@dns1005: END - running authdns-update
* 15:38 mutante: deleting mwmaint.discovery.wmnet DNS entry - the hosts behind it dont exist anymore
* 15:37 dzahn@dns1005: START - running authdns-update
* 15:24 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 15:24 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 15:21 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 15:20 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283024{{!}}Close Czech Wikinews (T421796)]] (duration: 06m 17s)
* 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92340 and previous config saved to /var/cache/conftool/dbconfig/20260505-151930-fceratto.json
* 15:16 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:16 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283024{{!}}Close Czech Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:14 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283024{{!}}Close Czech Wikinews (T421796)]]
* 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P92339 and previous config saved to /var/cache/conftool/dbconfig/20260505-150921-fceratto.json
* 15:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283003{{!}}Close Tamil Wikinews (T421796)]] (duration: 07m 06s)
* 15:07 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:04 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:03 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1283003{{!}}Close Tamil Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1283003{{!}}Close Tamil Wikinews (T421796)]]
* 14:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1283002{{!}}fix: wrong property name action_data (T425425)]] (duration: 07m 48s)
* 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P92338 and previous config saved to /var/cache/conftool/dbconfig/20260505-145913-fceratto.json
* 14:58 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 14:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 14:55 urbanecm@deploy1003: urbanecm: Continuing with deployment
* 14:53 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1283002{{!}}fix: wrong property name action_data (T425425)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92337 and previous config saved to /var/cache/conftool/dbconfig/20260505-145231-fceratto.json
* 14:51 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1283002{{!}}fix: wrong property name action_data (T425425)]]
* 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92336 and previous config saved to /var/cache/conftool/dbconfig/20260505-144905-fceratto.json
* 14:44 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1004.eqiad.wmnet with OS trixie
* 14:42 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P92335 and previous config saved to /var/cache/conftool/dbconfig/20260505-144223-fceratto.json
* 14:42 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 14:41 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 14:41 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92334 and previous config saved to /var/cache/conftool/dbconfig/20260505-144029-fceratto.json
* 14:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance
* 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92333 and previous config saved to /var/cache/conftool/dbconfig/20260505-143958-fceratto.json
* 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P92332 and previous config saved to /var/cache/conftool/dbconfig/20260505-143214-fceratto.json
* 14:30 jmm@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=config-master,name=eqiad
* 14:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P92331 and previous config saved to /var/cache/conftool/dbconfig/20260505-142949-fceratto.json
* 14:28 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1004.eqiad.wmnet with reason: host reimage
* 14:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM config-master1001.eqiad.wmnet
* 14:25 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1004.eqiad.wmnet with reason: host reimage
* 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM config-master1001.eqiad.wmnet
* 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92329 and previous config saved to /var/cache/conftool/dbconfig/20260505-142206-fceratto.json
* 14:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P92328 and previous config saved to /var/cache/conftool/dbconfig/20260505-141941-fceratto.json
* 14:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 14:11 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1004
* 14:10 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1004
* 14:10 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1004.eqiad.wmnet with OS trixie
* 14:09 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1015.eqiad.wmnet
* 14:09 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:09 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:09 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92327 and previous config saved to /var/cache/conftool/dbconfig/20260505-140933-fceratto.json
* 14:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 14:07 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 14:07 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 14:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 14:05 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 14:05 jmm@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=config-master,name=eqiad
* 14:05 jmm@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=config-master,name=codfw
* 14:04 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync
* 14:04 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync
* 14:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 14:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 14:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 14:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 14:03 Lucas_WMDE: UTC afternoon backport+config window done
* 14:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM config-master2001.codfw.wmnet
* 14:02 jasmine@cumin2002: END (FAIL) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=99) Change Confluent distribution for Kafka A:kafka-main-codfw cluster: Change Confluent distribution.
* 14:01 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1015.eqiad.wmnet
* 14:01 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1014.eqiad.wmnet
* 14:01 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:01 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:01 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92326 and previous config saved to /var/cache/conftool/dbconfig/20260505-140047-fceratto.json
* 14:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance
* 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92325 and previous config saved to /var/cache/conftool/dbconfig/20260505-140016-fceratto.json
* 13:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1227: Repooling
* 13:59 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:59 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM config-master2001.codfw.wmnet
* 13:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:55 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 13:54 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282988{{!}}Close Portuguese Wikinews (T421796)]] (duration: 06m 22s)
* 13:50 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1014.eqiad.wmnet
* 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P92323 and previous config saved to /var/cache/conftool/dbconfig/20260505-135008-fceratto.json
* 13:50 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 13:49 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282988{{!}}Close Portuguese Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:49 jmm@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=config-master,name=codfw
* 13:47 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1011.eqiad.wmnet
* 13:47 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:47 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1011.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:47 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282988{{!}}Close Portuguese Wikinews (T421796)]]
* 13:47 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1011.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92321 and previous config saved to /var/cache/conftool/dbconfig/20260505-134522-fceratto.json
* 13:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
* 13:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1227: Repooling
* 13:44 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 13:43 jasmine@cumin2002: START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-main-codfw cluster: Change Confluent distribution.
* 13:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92319 and previous config saved to /var/cache/conftool/dbconfig/20260505-134257-fceratto.json
* 13:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P92318 and previous config saved to /var/cache/conftool/dbconfig/20260505-134000-fceratto.json
* 13:37 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1011.eqiad.wmnet
* 13:37 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts aqs1010.eqiad.wmnet
* 13:37 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:37 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1010.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:37 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: aqs1010.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1003&quot;
* 13:33 eevans@cumin1003: START - Cookbook sre.dns.netbox
* 13:30 Msz2001: UTC afternoon backport window done
* 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92317 and previous config saved to /var/cache/conftool/dbconfig/20260505-132952-fceratto.json
* 13:27 eevans@cumin1003: START - Cookbook sre.hosts.decommission for hosts aqs1010.eqiad.wmnet
* 13:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 13:23 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 13:23 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270882{{!}}Remove temporary `wgOAuth2UsePrefixedSub` feature flag (T417690)]], [[gerrit:1271969{{!}}Move privileged global and local group handling to WikimediaCustomizations (T418507)]], [[gerrit:1281964{{!}}Add Akan (ak) to wmgExtraLanguageNames by default (T333765 T425256)]] (duration: 08m 37s)
* 13:23 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 13:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on dborch1002.wikimedia.org with reason: [[phab:T416582|T416582]]
* 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92316 and previous config saved to /var/cache/conftool/dbconfig/20260505-132002-fceratto.json
* 13:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance
* 13:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92315 and previous config saved to /var/cache/conftool/dbconfig/20260505-131931-fceratto.json
* 13:19 mszwarc@deploy1003: mszwarc, jhsoby, matmarex, d3r1ck01: Continuing with deployment
* 13:16 mszwarc@deploy1003: mszwarc, jhsoby, matmarex, d3r1ck01: Backport for [[gerrit:1270882{{!}}Remove temporary `wgOAuth2UsePrefixedSub` feature flag (T417690)]], [[gerrit:1271969{{!}}Move privileged global and local group handling to WikimediaCustomizations (T418507)]], [[gerrit:1281964{{!}}Add Akan (ak) to wmgExtraLanguageNames by default (T333765 T425256)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug
* 13:15 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1270882{{!}}Remove temporary `wgOAuth2UsePrefixedSub` feature flag (T417690)]], [[gerrit:1271969{{!}}Move privileged global and local group handling to WikimediaCustomizations (T418507)]], [[gerrit:1281964{{!}}Add Akan (ak) to wmgExtraLanguageNames by default (T333765 T425256)]]
* 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet
* 13:11 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282850{{!}}Switch 'autoconfirmed' to use APCOND_AGE_FROM_EDIT on certain wikis (T418484)]] (duration: 07m 55s)
* 13:11 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 13:11 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet
* 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P92314 and previous config saved to /var/cache/conftool/dbconfig/20260505-130923-fceratto.json
* 13:07 mszwarc@deploy1003: mszwarc: Continuing with deployment
* 13:05 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1282850{{!}}Switch 'autoconfirmed' to use APCOND_AGE_FROM_EDIT on certain wikis (T418484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1282850{{!}}Switch 'autoconfirmed' to use APCOND_AGE_FROM_EDIT on certain wikis (T418484)]]
* 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P92313 and previous config saved to /var/cache/conftool/dbconfig/20260505-125915-fceratto.json
* 12:56 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282970{{!}}Close Esperanto Wikinews (T421796)]] (duration: 07m 23s)
* 12:52 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 12:50 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282970{{!}}Close Esperanto Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:49 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282970{{!}}Close Esperanto Wikinews (T421796)]]
* 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92312 and previous config saved to /var/cache/conftool/dbconfig/20260505-124907-fceratto.json
* 12:44 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1280226{{!}}loggedOutWarning: instrument browser navigation and tab close (T421518)]] (duration: 03m 56s)
* 12:43 sgimeno@deploy1003: sgimeno: Continuing with deployment
* 12:42 moritzm: installing node-tar security updates
* 12:41 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1280226{{!}}loggedOutWarning: instrument browser navigation and tab close (T421518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:40 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1280226{{!}}loggedOutWarning: instrument browser navigation and tab close (T421518)]]
* 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2240 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92311 and previous config saved to /var/cache/conftool/dbconfig/20260505-124041-fceratto.json
* 12:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance
* 12:36 moritzm: installing imagemagick security updates
* 12:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2239.codfw.wmnet with reason: Maintenance
* 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92310 and previous config saved to /var/cache/conftool/dbconfig/20260505-123411-fceratto.json
* 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply
* 12:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:26 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P92309 and previous config saved to /var/cache/conftool/dbconfig/20260505-122404-fceratto.json
* 12:23 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:23 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P92308 and previous config saved to /var/cache/conftool/dbconfig/20260505-121352-fceratto.json
* 12:04 moritzm: installing postgresql-13 security updates
* 12:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92307 and previous config saved to /var/cache/conftool/dbconfig/20260505-120344-fceratto.json
* 11:57 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282960{{!}}Close Shan Wikinews (T421796)]] (duration: 06m 13s)
* 11:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd2005.codfw.wmnet
* 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92306 and previous config saved to /var/cache/conftool/dbconfig/20260505-115535-fceratto.json
* 11:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance
* 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92305 and previous config saved to /var/cache/conftool/dbconfig/20260505-115503-fceratto.json
* 11:53 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 11:53 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282960{{!}}Close Shan Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:52 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd2005.codfw.wmnet
* 11:51 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282960{{!}}Close Shan Wikinews (T421796)]]
* 11:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282955{{!}}Close Norwegian Wikinews (T421796)]] (duration: 09m 21s)
* 11:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd2004.codfw.wmnet
* 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P92304 and previous config saved to /var/cache/conftool/dbconfig/20260505-114455-fceratto.json
* 11:43 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 11:43 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd2004.codfw.wmnet
* 11:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd2003.codfw.wmnet
* 11:39 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282955{{!}}Close Norwegian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd2003.codfw.wmnet
* 11:38 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282955{{!}}Close Norwegian Wikinews (T421796)]]
* 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P92303 and previous config saved to /var/cache/conftool/dbconfig/20260505-113446-fceratto.json
* 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92302 and previous config saved to /var/cache/conftool/dbconfig/20260505-112449-fceratto.json
* 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92301 and previous config saved to /var/cache/conftool/dbconfig/20260505-112438-fceratto.json
* 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92300 and previous config saved to /var/cache/conftool/dbconfig/20260505-111616-fceratto.json
* 11:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance
* 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92299 and previous config saved to /var/cache/conftool/dbconfig/20260505-111545-fceratto.json
* 11:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P92298 and previous config saved to /var/cache/conftool/dbconfig/20260505-111435-fceratto.json
* 11:10 moritzm: installing ca-certificates updates from bookworm point release
* 11:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: after reimage to trixie
* 11:07 moritzm: installing multipart bugfix updates from bookworm point release
* 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P92296 and previous config saved to /var/cache/conftool/dbconfig/20260505-110537-fceratto.json
* 11:05 ayounsi@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 11:05 ayounsi@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs4009*&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P92295 and previous config saved to /var/cache/conftool/dbconfig/20260505-110427-fceratto.json
* 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1174: after reimage to trixie
* 10:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P92293 and previous config saved to /var/cache/conftool/dbconfig/20260505-105529-fceratto.json
* 10:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92291 and previous config saved to /var/cache/conftool/dbconfig/20260505-105419-fceratto.json
* 10:50 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
* 10:50 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
* 10:50 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
* 10:50 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
* 10:49 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
* 10:49 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
* 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92290 and previous config saved to /var/cache/conftool/dbconfig/20260505-104521-fceratto.json
* 10:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92288 and previous config saved to /var/cache/conftool/dbconfig/20260505-104032-fceratto.json
* 10:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92286 and previous config saved to /var/cache/conftool/dbconfig/20260505-103702-fceratto.json
* 10:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance
* 10:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92285 and previous config saved to /var/cache/conftool/dbconfig/20260505-103632-fceratto.json
* 10:32 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 10:29 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P92283 and previous config saved to /var/cache/conftool/dbconfig/20260505-102623-fceratto.json
* 10:24 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 10:24 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2221: after reimage to trixie
* 10:24 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 10:23 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 10:23 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 10:23 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 10:22 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2221.codfw.wmnet with OS trixie
* 10:17 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 10:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P92281 and previous config saved to /var/cache/conftool/dbconfig/20260505-101616-fceratto.json
* 10:14 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1174: after reimage to trixie
* 09:42 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 09:41 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 09:39 jelto@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 09:38 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P92271 and previous config saved to /var/cache/conftool/dbconfig/20260505-093703-fceratto.json
* 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92270 and previous config saved to /var/cache/conftool/dbconfig/20260505-093619-fceratto.json
* 09:36 jelto@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 09:35 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92269 and previous config saved to /var/cache/conftool/dbconfig/20260505-093305-fceratto.json
* 09:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 09:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 09:30 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1174.eqiad.wmnet with OS trixie
* 09:30 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2221.codfw.wmnet with OS trixie
* 09:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 09:30 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 09:29 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1174: Reimage to Trixie
* 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: Reimage to Trixie
* 09:29 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 09:28 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1174: Reimage to Trixie
* 09:28 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2221: Reimage to Trixie
* 09:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1174.eqiad.wmnet with reason: Reimage to Trixie
* 09:28 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2221.codfw.wmnet with reason: Reimage to Trixie
* 09:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92265 and previous config saved to /var/cache/conftool/dbconfig/20260505-092654-fceratto.json
* 09:26 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 09:25 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92264 and previous config saved to /var/cache/conftool/dbconfig/20260505-092431-fceratto.json
* 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92263 and previous config saved to /var/cache/conftool/dbconfig/20260505-091808-fceratto.json
* 09:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance
* 09:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P92262 and previous config saved to /var/cache/conftool/dbconfig/20260505-091423-fceratto.json
* 09:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2199.codfw.wmnet with reason: Maintenance
* 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92260 and previous config saved to /var/cache/conftool/dbconfig/20260505-091254-fceratto.json
* 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P92259 and previous config saved to /var/cache/conftool/dbconfig/20260505-090415-fceratto.json
* 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P92258 and previous config saved to /var/cache/conftool/dbconfig/20260505-090246-fceratto.json
* 08:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2209: after reimage to trixie
* 08:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92256 and previous config saved to /var/cache/conftool/dbconfig/20260505-085407-fceratto.json
* 08:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2209.codfw.wmnet with OS trixie
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P92255 and previous config saved to /var/cache/conftool/dbconfig/20260505-085238-fceratto.json
* 08:50 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 08:50 moritzm: installing augeas security updates
* 08:49 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) install5004.wikimedia.org on all recursors
* 08:48 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 08:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:48 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Remove records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92254 and previous config saved to /var/cache/conftool/dbconfig/20260505-084616-fceratto.json
* 08:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 08:42 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92253 and previous config saved to /var/cache/conftool/dbconfig/20260505-084231-fceratto.json
* 08:41 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:40 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:38 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:37 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:37 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 08:37 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 08:35 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 08:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 08:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 08:34 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 13 hosts with reason: switches replacement
* 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92252 and previous config saved to /var/cache/conftool/dbconfig/20260505-083356-fceratto.json
* 08:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
* 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92251 and previous config saved to /var/cache/conftool/dbconfig/20260505-083326-fceratto.json
* 08:32 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:32 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 08:32 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 08:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2209.codfw.wmnet with reason: host reimage
* 08:29 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) install5004.wikimedia.org on all recursors
* 08:28 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:28 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 08:24 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P92250 and previous config saved to /var/cache/conftool/dbconfig/20260505-082318-fceratto.json
* 08:22 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: after reimage to trixie
* 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2209.codfw.wmnet with reason: host reimage
* 08:16 zabe: zabe@deploy1003:~$ foreachwiki refreshImageMetadata --broken-only --mediatype AUDIO --mime audio/x-flac # [[phab:T414641|T414641]]
* 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1170: after reimage to trixie
* 08:14 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P92247 and previous config saved to /var/cache/conftool/dbconfig/20260505-081309-fceratto.json
* 08:08 zabe: zabe@deploy1003:~$ foreachwiki refreshImageMetadata --broken-only --mediatype AUDIO --mime audio/flac # [[phab:T414641|T414641]]
* 08:05 ayounsi@dns1004: END - running authdns-update
* 08:03 ayounsi@dns1004: START - running authdns-update
* 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92245 and previous config saved to /var/cache/conftool/dbconfig/20260505-080301-fceratto.json
* 08:02 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2209.codfw.wmnet with OS trixie
* 08:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ulsfo includes - ayounsi@cumin1003&quot;
* 08:01 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ulsfo includes - ayounsi@cumin1003&quot;
* 08:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: Reimage to Trixie
* 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2209: Reimage to Trixie
* 08:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2209.codfw.wmnet with reason: Reimage to Trixie
* 07:58 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 07:57 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2209 [[phab:T424864|T424864]]', diff saved to https://phabricator.wikimedia.org/P92243 and previous config saved to /var/cache/conftool/dbconfig/20260505-075746-marostegui.json
* 07:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2205 to s3 primary [[phab:T424864|T424864]]', diff saved to https://phabricator.wikimedia.org/P92242 and previous config saved to /var/cache/conftool/dbconfig/20260505-075654-marostegui.json
* 07:55 awight: EU morning deployment was fun
* 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92241 and previous config saved to /var/cache/conftool/dbconfig/20260505-075416-fceratto.json
* 07:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
* 07:52 marostegui: Starting s3 codfw failover from db2209 to db2205 - [[phab:T424864|T424864]]
* 07:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2205 with weight 0 [[phab:T424864|T424864]]', diff saved to https://phabricator.wikimedia.org/P92239 and previous config saved to /var/cache/conftool/dbconfig/20260505-075156-marostegui.json
* 07:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 [[phab:T424864|T424864]]
* 07:50 zabe: zabe@deploy1003:~$ foreachwiki refreshImageMetadata --force --mediatype AUDIO --mime audio/midi # [[phab:T414645|T414645]]
* 07:45 zabe: zabe@deploy1003:~$ mwscript namespaceDupes.php scnwiki --fix # [[phab:T425378|T425378]]
* 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2222: after reimage to trixie
* 07:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2222.codfw.wmnet with OS trixie
* 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1170: after reimage to trixie
* 07:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1170.eqiad.wmnet with OS trixie
* 07:11 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281967{{!}}zhwikinews: (2/2) revert 20th anniversary logo change (assets) (T420165)]] (duration: 06m 43s)
* 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2222.codfw.wmnet with reason: host reimage
* 07:07 awight@deploy1003: awight, 1f616emo: Continuing with deployment
* 07:06 awight@deploy1003: awight, 1f616emo: Backport for [[gerrit:1281967{{!}}zhwikinews: (2/2) revert 20th anniversary logo change (assets) (T420165)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1281967{{!}}zhwikinews: (2/2) revert 20th anniversary logo change (assets) (T420165)]]
* 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1170.eqiad.wmnet with reason: host reimage
* 07:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 07:03 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 07:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2222.codfw.wmnet with reason: host reimage
* 07:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: after reimage to trixie
* 06:58 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 06:58 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 06:58 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1170.eqiad.wmnet with reason: host reimage
* 06:44 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1170.eqiad.wmnet with OS trixie
* 06:44 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2222.codfw.wmnet with OS trixie
* 06:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1170: Reimage to Trixie
* 06:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: Reimage to Trixie
* 06:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1170: Reimage to Trixie
* 06:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Reimage to Trixie
* 06:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2222: Reimage to Trixie
* 06:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2222.codfw.wmnet with reason: Reimage to Trixie
* 06:14 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1156: after reimage to trixie
* 06:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1156.eqiad.wmnet with OS trixie
* 05:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1156.eqiad.wmnet with reason: host reimage
* 05:46 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1156.eqiad.wmnet with reason: host reimage
* 05:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;patterns_as_inline_patterns - oblivian@cumin1003&quot;
* 05:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: patterns_as_inline_patterns - oblivian@cumin1003
* 05:42 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: patterns_as_inline_patterns - oblivian@cumin1003
* 05:42 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;patterns_as_inline_patterns - oblivian@cumin1003&quot;
* 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1156.eqiad.wmnet with OS trixie
* 05:31 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1156: Reimage to Trixie
* 05:30 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1156: Reimage to Trixie
* 05:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Reimage to Trixie
* 05:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: Sanitarium s2 master: reimage to Debian Trixie
* 04:03 mwpresync@deploy1003: Pruned MediaWiki: 1.46.0-wmf.23 (duration: 03m 12s)
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 39s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:21 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 01:21 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add dns infor for new switches - pt1979@cumin2002&quot;
* 01:21 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add dns infor for new switches - pt1979@cumin2002&quot;
* 01:16 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 00:16 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282434{{!}}Close Catalan Wikinews (T421796)]] (duration: 06m 50s)
* 00:11 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 00:10 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282434{{!}}Close Catalan Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:09 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282434{{!}}Close Catalan Wikinews (T421796)]]

== 2026-05-04 ==
* 23:48 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282432{{!}}Close Bosnian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:46 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282432{{!}}Close Bosnian Wikinews (T421796)]]
* 23:14 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282410{{!}}Close Hebrew Wikinews (T421796)]] (duration: 06m 45s)
* 23:10 ladsgroup@deploy1003: neriah, ladsgroup: Continuing with deployment
* 23:09 ladsgroup@deploy1003: neriah, ladsgroup: Backport for [[gerrit:1282410{{!}}Close Hebrew Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282410{{!}}Close Hebrew Wikinews (T421796)]]
* 22:08 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 22:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 22:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 22:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 21:43 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 21:42 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 21:32 cwhite@deploy1003: Finished deploy [statsv/statsv@152de49]: fix logging (duration: 00m 11s)
* 21:32 cwhite@deploy1003: Started deploy [statsv/statsv@152de49]: fix logging
* 21:20 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1276432{{!}}Enable Hebrew keyboard DWIM for namespace resolution on hewikis (T412468)]] (duration: 11m 20s)
* 21:16 cjming@deploy1003: cjming, neriah: Continuing with deployment
* 21:10 cjming@deploy1003: cjming, neriah: Backport for [[gerrit:1276432{{!}}Enable Hebrew keyboard DWIM for namespace resolution on hewikis (T412468)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:09 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1276432{{!}}Enable Hebrew keyboard DWIM for namespace resolution on hewikis (T412468)]]
* 20:38 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282385{{!}}Revert^2 &quot;Use js promise for email confirmation banner&quot;]] (duration: 22m 19s)
* 20:34 cjming@deploy1003: mmartorana, cjming: Continuing with deployment
* 20:18 cjming@deploy1003: mmartorana, cjming: Backport for [[gerrit:1282385{{!}}Revert^2 &quot;Use js promise for email confirmation banner&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:16 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1282385{{!}}Revert^2 &quot;Use js promise for email confirmation banner&quot;]]
* 20:11 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277667{{!}}Enable the reading list beta feature survey on all wikipedias (T421776)]] (duration: 07m 21s)
* 20:07 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1005.eqiad.wmnet with OS trixie
* 20:06 toyofuku@deploy1003: toyofuku: Continuing with deployment
* 20:05 toyofuku@deploy1003: toyofuku: Backport for [[gerrit:1277667{{!}}Enable the reading list beta feature survey on all wikipedias (T421776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:03 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1277667{{!}}Enable the reading list beta feature survey on all wikipedias (T421776)]]
* 19:51 ayounsi@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) asw1-22-ulsfo.wikimedia.org on all recursors
* 19:50 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache asw1-22-ulsfo.wikimedia.org on all recursors
* 19:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: asw1-22-ulsfo - ayounsi@cumin1003&quot;
* 19:49 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: asw1-22-ulsfo - ayounsi@cumin1003&quot;
* 19:48 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1005.eqiad.wmnet with reason: host reimage
* 19:44 ayounsi@cumin1003: START - Cookbook sre.dns.netbox
* 19:42 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1005.eqiad.wmnet with reason: host reimage
* 19:40 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:37 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:28 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: ongoing troubleshooting
* 19:27 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1005
* 19:27 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging1005
* 19:27 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1005.eqiad.wmnet with OS trixie
* 19:23 root@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
* 19:23 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:23 root@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
* 19:23 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - [[phab:T424852|T424852]]
* 19:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 19:06 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 18:59 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282407{{!}}Close Limburgish Wikinews (T421796)]] (duration: 06m 16s)
* 18:55 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 18:55 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282407{{!}}Close Limburgish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:53 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282407{{!}}Close Limburgish Wikinews (T421796)]]
* 18:31 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282405{{!}}Close Albanian Wikinews (T421796)]] (duration: 09m 17s)
* 18:27 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 18:23 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282405{{!}}Close Albanian Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:22 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282405{{!}}Close Albanian Wikinews (T421796)]]
* 18:11 dancy@deploy1003: Finished scap sync-world: testing (duration: 02m 04s)
* 18:11 dancy@deploy1003: dancy: Rolling back deployment
* 18:10 dancy@deploy1003: dancy: testing synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:09 dancy@deploy1003: Started scap sync-world: testing
* 18:08 dancy@deploy1003: Installation of scap version &quot;4.260.0&quot; completed for 2 hosts
* 18:06 dancy@deploy1003: Installing scap version &quot;4.260.0&quot; for 2 host(s)
* 17:49 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:47 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:47 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:41 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 17:31 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 16:40 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:39 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:34 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 16:33 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:33 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 16:33 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 16:04 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282384{{!}}Close Greek Wikinews (T421796)]] (duration: 06m 19s)
* 16:00 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 16:00 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282384{{!}}Close Greek Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:58 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282384{{!}}Close Greek Wikinews (T421796)]]
* 15:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92224 and previous config saved to /var/cache/conftool/dbconfig/20260504-155514-fceratto.json
* 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92223 and previous config saved to /var/cache/conftool/dbconfig/20260504-154506-fceratto.json
* 15:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282060{{!}}Make errorpages responsive]] (duration: 06m 59s)
* 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92222 and previous config saved to /var/cache/conftool/dbconfig/20260504-153458-fceratto.json
* 15:34 ladsgroup@deploy1003: ladsgroup, chlod: Continuing with deployment
* 15:33 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 39 hosts with reason: switches replacement
* 15:33 ladsgroup@deploy1003: ladsgroup, chlod: Backport for [[gerrit:1282060{{!}}Make errorpages responsive]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:32 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: sync
* 15:32 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: sync
* 15:31 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282060{{!}}Make errorpages responsive]]
* 15:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92221 and previous config saved to /var/cache/conftool/dbconfig/20260504-152449-fceratto.json
* 15:22 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92220 and previous config saved to /var/cache/conftool/dbconfig/20260504-152238-fceratto.json
* 15:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance
* 15:20 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:17 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 15:17 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 15:16 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:15 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:13 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host install5004.wikimedia.org
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 15:13 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92219 and previous config saved to /var/cache/conftool/dbconfig/20260504-151238-fceratto.json
* 15:10 papaul: ongoing switch refresh in ULSFO
* 15:10 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 15:10 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 15:06 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 15:05 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282381{{!}}Close Gun Wikinews (T421796)]] (duration: 06m 45s)
* 15:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92218 and previous config saved to /var/cache/conftool/dbconfig/20260504-150230-fceratto.json
* 15:01 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 15:00 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1282381{{!}}Close Gun Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:58 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1282381{{!}}Close Gun Wikinews (T421796)]]
* 14:58 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2001.codfw.wmnet with OS trixie
* 14:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92217 and previous config saved to /var/cache/conftool/dbconfig/20260504-145222-fceratto.json
* 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92216 and previous config saved to /var/cache/conftool/dbconfig/20260504-144213-fceratto.json
* 14:41 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts
* 14:41 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 7 hosts
* 14:39 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2001.codfw.wmnet with reason: host reimage
* 14:34 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2001.codfw.wmnet with reason: host reimage
* 14:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92215 and previous config saved to /var/cache/conftool/dbconfig/20260504-143334-fceratto.json
* 14:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2229.codfw.wmnet with reason: Maintenance
* 14:30 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cr[3-4]-ulsfo IPv6,cr[3-4]-ulsfo.mgmt,mr1-ulsfo IPv6 with reason: switch refresh
* 14:28 pt1979@cumin1003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 4:00:00 on cr[3-4]-ulsfo IPV6,cr[3-4]-ulsfo.mgmt,mr1-ulsfo IPV6 with reason: switch refresh
* 14:25 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on asw2-ulsfo,cr[3-4]-ulsfo,mr1-ulsfo with reason: switch refresh
* 14:16 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2001
* 14:16 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2001
* 14:13 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2001
* 14:13 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2001.codfw.wmnet 94.0.192.10.in-addr.arpa 4.9.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:13 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2001.codfw.wmnet 94.0.192.10.in-addr.arpa 4.9.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:13 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:13 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2001 - herron@cumin1003&quot;
* 14:13 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2001 - herron@cumin1003&quot;
* 14:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92214 and previous config saved to /var/cache/conftool/dbconfig/20260504-141113-fceratto.json
* 14:07 herron@cumin1003: START - Cookbook sre.dns.netbox
* 14:04 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2001
* 14:04 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2001.codfw.wmnet with OS trixie
* 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P92213 and previous config saved to /var/cache/conftool/dbconfig/20260504-140105-fceratto.json
* 14:00 slyngshede@cumin1003: conftool action : set/pooled=no; selector: cluster=dnsbox,dc=ulsfo [reason: ulsfo switch refresh [[phab:T408892|T408892]]]
* 14:00 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 14:00 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool ulsfo [reason: New switch configuration, [[phab:T408892|T408892]]]
* 13:59 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1282354{{!}}ArticleGuidance: enable on simple english (T425351)]] (duration: 06m 22s)
* 13:57 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) install5004.wikimedia.org on all recursors
* 13:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache install5004.wikimedia.org on all recursors
* 13:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:56 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 13:56 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM install5004.wikimedia.org - jmm@cumin2002&quot;
* 13:55 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 13:55 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 13:55 sbisson@deploy1003: sbisson: Continuing with deployment
* 13:55 sbisson@deploy1003: sbisson: Backport for [[gerrit:1282354{{!}}ArticleGuidance: enable on simple english (T425351)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:54 dcausse: [[phab:T425301|T425301]]: stopping writes again on cloudelastic, cluster unstable
* 13:53 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1282354{{!}}ArticleGuidance: enable on simple english (T425351)]]
* 13:52 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:52 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host install5004.wikimedia.org
* 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P92212 and previous config saved to /var/cache/conftool/dbconfig/20260504-135056-fceratto.json
* 13:50 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281965{{!}}zhwikinews: (1/2) revert 20th anniversary logo change (config) (T420165)]] (duration: 07m 30s)
* 13:46 sbisson@deploy1003: 1f616emo, sbisson: Continuing with deployment
* 13:45 sbisson@deploy1003: 1f616emo, sbisson: Backport for [[gerrit:1281965{{!}}zhwikinews: (1/2) revert 20th anniversary logo change (config) (T420165)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:43 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1281965{{!}}zhwikinews: (1/2) revert 20th anniversary logo change (config) (T420165)]]
* 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92211 and previous config saved to /var/cache/conftool/dbconfig/20260504-134048-fceratto.json
* 13:40 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92210 and previous config saved to /var/cache/conftool/dbconfig/20260504-133039-fceratto.json
* 13:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92209 and previous config saved to /var/cache/conftool/dbconfig/20260504-133010-fceratto.json
* 13:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:23 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:23 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P92208 and previous config saved to /var/cache/conftool/dbconfig/20260504-132002-fceratto.json
* 13:13 moritzm: installing jaraco.context security updates
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum5004.eqsin.wmnet
* 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host durum5004.eqsin.wmnet with OS bookworm
* 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P92207 and previous config saved to /var/cache/conftool/dbconfig/20260504-130953-fceratto.json
* 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92206 and previous config saved to /var/cache/conftool/dbconfig/20260504-125945-fceratto.json
* 12:59 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 12:59 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 12:59 dcausse: [[phab:T425301|T425301]]: resuming writes on cloudelastic
* 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92205 and previous config saved to /var/cache/conftool/dbconfig/20260504-125247-fceratto.json
* 12:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92204 and previous config saved to /var/cache/conftool/dbconfig/20260504-125219-fceratto.json
* 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 12:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on durum5004.eqsin.wmnet with reason: host reimage
* 12:45 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on durum5004.eqsin.wmnet with reason: host reimage
* 12:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P92203 and previous config saved to /var/cache/conftool/dbconfig/20260504-124210-fceratto.json
* 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P92202 and previous config saved to /var/cache/conftool/dbconfig/20260504-123203-fceratto.json
* 12:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92201 and previous config saved to /var/cache/conftool/dbconfig/20260504-122155-fceratto.json
* 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1203 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92200 and previous config saved to /var/cache/conftool/dbconfig/20260504-121441-fceratto.json
* 12:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
* 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92199 and previous config saved to /var/cache/conftool/dbconfig/20260504-121424-fceratto.json
* 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P92198 and previous config saved to /var/cache/conftool/dbconfig/20260504-120416-fceratto.json
* 12:03 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host durum5004.eqsin.wmnet with OS bookworm
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) durum5004.eqsin.wmnet on all recursors
* 11:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache durum5004.eqsin.wmnet on all recursors
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P92197 and previous config saved to /var/cache/conftool/dbconfig/20260504-115408-fceratto.json
* 11:51 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5004.eqsin.wmnet - jmm@cumin2002&quot;
* 11:47 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 11:47 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host durum5004.eqsin.wmnet
* 11:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum5003.eqsin.wmnet
* 11:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host durum5003.eqsin.wmnet with OS bookworm
* 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92196 and previous config saved to /var/cache/conftool/dbconfig/20260504-114400-fceratto.json
* 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1193 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92195 and previous config saved to /var/cache/conftool/dbconfig/20260504-113620-fceratto.json
* 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
* 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92194 and previous config saved to /var/cache/conftool/dbconfig/20260504-113550-fceratto.json
* 11:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: after reimage to trixie
* 11:26 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on durum5003.eqsin.wmnet with reason: host reimage
* 11:25 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on durum5003.eqsin.wmnet with reason: host reimage
* 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P92192 and previous config saved to /var/cache/conftool/dbconfig/20260504-112542-fceratto.json
* 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P92191 and previous config saved to /var/cache/conftool/dbconfig/20260504-111534-fceratto.json
* 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92189 and previous config saved to /var/cache/conftool/dbconfig/20260504-110526-fceratto.json
* 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2187: repool after maintenance
* 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92187 and previous config saved to /var/cache/conftool/dbconfig/20260504-105808-fceratto.json
* 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92186 and previous config saved to /var/cache/conftool/dbconfig/20260504-105739-fceratto.json
* 10:48 moritzm: installing bash updates from trixie point release
* 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P92184 and previous config saved to /var/cache/conftool/dbconfig/20260504-104731-fceratto.json
* 10:42 moritzm: installing postgresql-17 security updates
* 10:42 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1162: after reimage to trixie
* 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1162.eqiad.wmnet with OS trixie
* 10:38 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS bookworm
* 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P92181 and previous config saved to /var/cache/conftool/dbconfig/20260504-103723-fceratto.json
* 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:35 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) durum5003.eqsin.wmnet on all recursors
* 10:34 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache durum5003.eqsin.wmnet on all recursors
* 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:34 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum5003.eqsin.wmnet - jmm@cumin2002&quot;
* 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92179 and previous config saved to /var/cache/conftool/dbconfig/20260504-102715-fceratto.json
* 10:26 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host durum5003.eqsin.wmnet
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1178 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92178 and previous config saved to /var/cache/conftool/dbconfig/20260504-101855-fceratto.json
* 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
* 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92177 and previous config saved to /var/cache/conftool/dbconfig/20260504-101826-fceratto.json
* 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2187: repool after maintenance
* 10:16 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
* 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
* 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P92174 and previous config saved to /var/cache/conftool/dbconfig/20260504-100818-fceratto.json
* 10:02 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1162.eqiad.wmnet with OS trixie
* 10:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reimage to Trixie
* 10:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reimage to Trixie
* 10:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Reimage to Trixie
* 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P92172 and previous config saved to /var/cache/conftool/dbconfig/20260504-095810-fceratto.json
* 09:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org
* 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92171 and previous config saved to /var/cache/conftool/dbconfig/20260504-094802-fceratto.json
* 09:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1177 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92170 and previous config saved to /var/cache/conftool/dbconfig/20260504-093938-fceratto.json
* 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92169 and previous config saved to /var/cache/conftool/dbconfig/20260504-093910-fceratto.json
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 09:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1182: after reimage to trixie
* 09:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P92167 and previous config saved to /var/cache/conftool/dbconfig/20260504-092902-fceratto.json
* 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P92165 and previous config saved to /var/cache/conftool/dbconfig/20260504-091853-fceratto.json
* 09:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2187: Fixing events
* 09:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2187: Fixing events
* 09:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2187.codfw.wmnet with reason: Checking events
* 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92163 and previous config saved to /var/cache/conftool/dbconfig/20260504-090845-fceratto.json
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1172 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92161 and previous config saved to /var/cache/conftool/dbconfig/20260504-085930-fceratto.json
* 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92160 and previous config saved to /var/cache/conftool/dbconfig/20260504-085912-fceratto.json
* 08:56 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 08:55 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 08:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1182: after reimage to trixie
* 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P92158 and previous config saved to /var/cache/conftool/dbconfig/20260504-084904-fceratto.json
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1008.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1007.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1006.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1005.eqiad.wmnet
* 08:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1004.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1003.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1002.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-worker1001.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-ctrl1002.eqiad.wmnet
* 08:42 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: tools-k8s-ctrl1001.eqiad.wmnet
* 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P92157 and previous config saved to /var/cache/conftool/dbconfig/20260504-083857-fceratto.json
* 08:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1182.eqiad.wmnet with OS trixie
* 08:32 moritzm: installing Linux 5.10.251-3 on bullseye hosts
* 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92156 and previous config saved to /var/cache/conftool/dbconfig/20260504-082849-fceratto.json
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1003.eqiad.wmnet
* 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T419961|T419961]])', diff saved to https://phabricator.wikimedia.org/P92155 and previous config saved to /var/cache/conftool/dbconfig/20260504-082024-fceratto.json
* 08:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 08:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 08:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1003.eqiad.wmnet
* 08:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1182.eqiad.wmnet with reason: host reimage
* 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1182.eqiad.wmnet with reason: host reimage
* 08:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
* 08:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
* 08:04 gkyziridis@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync
* 08:04 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277256{{!}}Add sva to wmgExtraLanguageNames (T407106)]] (duration: 07m 58s)
* 08:03 gkyziridis@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync
* 08:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 08:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 08:02 gkyziridis@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: sync
* 08:02 gkyziridis@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: sync
* 08:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
* 08:01 moritzm: installing Linux 6.1.170 on bookworm hosts
* 07:59 urbanecm@deploy1003: urbanecm, h2o: Continuing with deployment
* 07:57 urbanecm@deploy1003: urbanecm, h2o: Backport for [[gerrit:1277256{{!}}Add sva to wmgExtraLanguageNames (T407106)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1212: after reimage to trixie
* 07:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1277256{{!}}Add sva to wmgExtraLanguageNames (T407106)]]
* 07:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1182.eqiad.wmnet with OS trixie
* 07:51 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 07:51 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
* 07:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1182: Reimage to Trixie
* 07:47 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1182: Reimage to Trixie
* 07:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Reimage to Trixie
* 07:44 dcausse: [[phab:T425301|T425301]]: stopping writes on cloudelastic
* 07:44 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 07:44 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2147.codfw.wmnet
* 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2147.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 07:42 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: db2147.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003&quot;
* 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2149: after reimage to trixie
* 07:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1188: after reimage to trixie
* 07:38 moritzm: installing Linux 6.12.85 on trixie hosts
* 07:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-jumbo2003.codfw.wmnet
* 07:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
* 07:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-jumbo2003.codfw.wmnet
* 07:33 marostegui@cumin1003: START - Cookbook sre.dns.netbox
* 07:28 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2147.codfw.wmnet
* 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org
* 07:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test1005.wikimedia.org
* 07:11 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1212: after reimage to trixie
* 07:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1212.eqiad.wmnet with OS trixie
* 06:56 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2149: after reimage to trixie
* 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1188: after reimage to trixie
* 06:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1188.eqiad.wmnet with OS trixie
* 06:47 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2149.codfw.wmnet with OS trixie
* 06:43 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1212.eqiad.wmnet with reason: host reimage
* 06:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1212.eqiad.wmnet with reason: host reimage
* 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
* 06:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
* 06:21 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1212.eqiad.wmnet with OS trixie
* 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
* 06:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
* 06:11 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1212: Reimage to Trixie
* 06:11 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1212: Reimage to Trixie
* 06:11 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1212.eqiad.wmnet with reason: Reimage to Trixie
* 06:10 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: Sanitarium s3 master: reimage to Debian Trixie
* 06:09 marostegui: Reimage sanitarium master for s3, lag to be expected on wikireplicas for s3 [[phab:T424792|T424792]]
* 06:05 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1188.eqiad.wmnet with OS trixie
* 06:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1188: Reimage to Trixie
* 05:58 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1188: Reimage to Trixie
* 05:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Reimage to Trixie
* 05:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2149.codfw.wmnet with OS trixie
* 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2149: Reimage to Trixie
* 05:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2149: Reimage to Trixie
* 05:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2149.codfw.wmnet with reason: Reimage to Trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 36s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-05-03 ==
* 14:11 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281991{{!}}Disable uploads in scnwiki (T425278)]] (duration: 10m 51s)
* 14:05 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 14:04 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1281991{{!}}Disable uploads in scnwiki (T425278)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:00 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1281991{{!}}Disable uploads in scnwiki (T425278)]]
* 12:27 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281963{{!}}Remove Wikinews from installer's default main page]] (duration: 29m 22s)
* 11:58 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1281963{{!}}Remove Wikinews from installer's default main page]]

== 2026-05-02 ==
* 23:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281775{{!}}Uninstall DynamicPageList from wikis it's not used on (T425202)]] (duration: 06m 41s)
* 23:28 zabe@deploy1003: dreamyjazz, zabe: Continuing with deployment
* 23:27 zabe@deploy1003: dreamyjazz, zabe: Backport for [[gerrit:1281775{{!}}Uninstall DynamicPageList from wikis it's not used on (T425202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:26 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281775{{!}}Uninstall DynamicPageList from wikis it's not used on (T425202)]]
* 23:22 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281739{{!}}Uninstall DynamicPageList from officewiki (T425154)]] (duration: 07m 27s)
* 23:18 zabe@deploy1003: zabe, dreamyjazz: Continuing with deployment
* 23:17 zabe@deploy1003: zabe, dreamyjazz: Backport for [[gerrit:1281739{{!}}Uninstall DynamicPageList from officewiki (T425154)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:15 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1281739{{!}}Uninstall DynamicPageList from officewiki (T425154)]]
* 18:07 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie
* 18:07 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie
* 18:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host rdb2014.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host rdb2014.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:43 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2369.codfw.wmnet with OS trixie
* 17:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:36 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:18 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2369.codfw.wmnet with reason: host reimage
* 17:14 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2369.codfw.wmnet with reason: host reimage
* 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2374.codfw.wmnet with OS trixie
* 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:13 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:09 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2373.codfw.wmnet with OS trixie
* 17:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:09 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2372.codfw.wmnet with OS trixie
* 17:06 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:05 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:01 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2371.codfw.wmnet with OS trixie
* 17:01 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2370.codfw.wmnet with OS trixie
* 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 16:56 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2374.codfw.wmnet with reason: host reimage
* 16:56 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 16:52 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2373.codfw.wmnet with reason: host reimage
* 16:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 16:48 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2372.codfw.wmnet with reason: host reimage
* 16:44 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2374.codfw.wmnet with reason: host reimage
* 16:43 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2373.codfw.wmnet with reason: host reimage
* 16:42 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2372.codfw.wmnet with reason: host reimage
* 16:36 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2371.codfw.wmnet with reason: host reimage
* 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2370.codfw.wmnet with reason: host reimage
* 16:32 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2371.codfw.wmnet with reason: host reimage
* 16:31 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2374.codfw.wmnet with OS trixie
* 16:30 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2373.codfw.wmnet with OS trixie
* 16:29 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2372.codfw.wmnet with OS trixie
* 16:28 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2370.codfw.wmnet with reason: host reimage
* 16:20 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2371.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2371.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2371.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2370.codfw.wmnet with OS trixie
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2369.codfw.wmnet with OS trixie
* 16:08 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2366.codfw.wmnet with OS trixie
* 16:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2367.codfw.wmnet with OS trixie
* 15:59 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2368.codfw.wmnet with OS trixie
* 15:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 15:47 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 15:43 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 15:38 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 15:38 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 15:36 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 15:25 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2366.codfw.wmnet with OS trixie
* 15:24 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2367.codfw.wmnet with OS trixie
* 15:23 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2368.codfw.wmnet with OS trixie
* 12:02 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281747{{!}}Watchlist star: Revert popover/dialog changes (T425185)]] (duration: 13m 06s)
* 11:57 samtar@deploy1003: samtar: Continuing with deployment
* 11:50 samtar@deploy1003: samtar: Backport for [[gerrit:1281747{{!}}Watchlist star: Revert popover/dialog changes (T425185)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:49 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1281747{{!}}Watchlist star: Revert popover/dialog changes (T425185)]]
* 09:20 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 09:19 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2366.codfw.wmnet with OS trixie
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2368.codfw.wmnet with OS trixie
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2367.codfw.wmnet with OS trixie
* 02:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:52 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:23 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:17 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:12 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 31s)
* 02:02 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:58 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 01:54 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 01:50 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2368.codfw.wmnet with reason: host reimage
* 01:50 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2367.codfw.wmnet with reason: host reimage
* 01:49 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2366.codfw.wmnet with reason: host reimage
* 01:37 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2368.codfw.wmnet with OS trixie
* 01:37 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2367.codfw.wmnet with OS trixie
* 01:37 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2366.codfw.wmnet with OS trixie
* 01:35 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2364.codfw.wmnet with OS trixie
* 01:34 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:29 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:24 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2365.codfw.wmnet with OS trixie
* 01:24 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:23 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:21 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2363.codfw.wmnet with OS trixie
* 01:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 01:13 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2364.codfw.wmnet with reason: host reimage
* 01:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2365.codfw.wmnet with reason: host reimage
* 01:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2363.codfw.wmnet with reason: host reimage
* 00:58 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2365.codfw.wmnet with reason: host reimage
* 00:57 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2364.codfw.wmnet with reason: host reimage
* 00:57 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2363.codfw.wmnet with reason: host reimage
* 00:45 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2365.codfw.wmnet with OS trixie
* 00:45 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2364.codfw.wmnet with OS trixie
* 00:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2363.codfw.wmnet with OS trixie
* 00:09 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2362.codfw.wmnet with OS trixie
* 00:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:07 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2361.codfw.wmnet with OS trixie
* 00:05 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:02 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:01 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2360.codfw.wmnet with OS trixie
* 00:01 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 00:01 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;

== 2026-05-01 ==
* 23:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2362.codfw.wmnet with reason: host reimage
* 23:47 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2361.codfw.wmnet with reason: host reimage
* 23:45 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2360.codfw.wmnet with reason: host reimage
* 23:39 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2362.codfw.wmnet with reason: host reimage
* 23:39 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2361.codfw.wmnet with reason: host reimage
* 23:39 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2360.codfw.wmnet with reason: host reimage
* 23:27 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2362.codfw.wmnet with OS trixie
* 23:27 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2361.codfw.wmnet with OS trixie
* 23:26 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2360.codfw.wmnet with OS trixie
* 23:25 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2357.codfw.wmnet with OS trixie
* 23:25 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:25 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:22 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2359.codfw.wmnet with OS trixie
* 23:22 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:22 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:18 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2358.codfw.wmnet with OS trixie
* 23:18 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:18 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 23:08 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2357.codfw.wmnet with reason: host reimage
* 23:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2359.codfw.wmnet with reason: host reimage
* 23:02 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2358.codfw.wmnet with reason: host reimage
* 22:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2359.codfw.wmnet with reason: host reimage
* 22:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2357.codfw.wmnet with reason: host reimage
* 22:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2358.codfw.wmnet with reason: host reimage
* 22:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2359.codfw.wmnet with OS trixie
* 22:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2358.codfw.wmnet with OS trixie
* 22:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2357.codfw.wmnet with OS trixie
* 22:39 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:28 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:25 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:24 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:19 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:17 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:15 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:06 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 22:02 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:59 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:57 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:51 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:50 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:49 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:48 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:48 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:46 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:41 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:41 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2364.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:38 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2363.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:33 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2364.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:32 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:31 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2363.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2362.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:25 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2361.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:24 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2360.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:21 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2361.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:17 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2360.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:15 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2359.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:13 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2358.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2359.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2358.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2357.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 21:04 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2374
* 21:04 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2374
* 21:04 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2373
* 21:04 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2373
* 20:59 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2372
* 20:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2372
* 20:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2371
* 20:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2371
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2370
* 20:57 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2370
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2369
* 20:57 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2369
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2368
* 20:57 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2368
* 20:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2367
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2367
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2366
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2366
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2365
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2365
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2364
* 20:56 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2364
* 20:56 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2363
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2363
* 20:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2362
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2362
* 20:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2361
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2361
* 20:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2360
* 20:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2360
* 20:54 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2359
* 20:54 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2359
* 20:54 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2358
* 20:54 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2358
* 20:54 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2357
* 20:53 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2357
* 20:53 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:53 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2357 to codfw - jhancock@cumin2002&quot;
* 20:53 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2357 to codfw - jhancock@cumin2002&quot;
* 20:49 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 20:14 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2002.codfw.wmnet with OS trixie
* 20:06 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269440{{!}}Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441{{!}}Enable wgTrackMediaRequestProvenance on Commons (T414338)]] (duration: 15m 27s)
* 20:02 krinkle@deploy1003: krinkle: Continuing with deployment
* 19:54 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage
* 19:52 krinkle@deploy1003: krinkle: Backport for [[gerrit:1269440{{!}}Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441{{!}}Enable wgTrackMediaRequestProvenance on Commons (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:51 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1269440{{!}}Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441{{!}}Enable wgTrackMediaRequestProvenance on Commons (T414338)]]
* 19:49 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage
* 19:40 dancy@deploy1003: Finished scap sync-world: testing [[phab:T317405|T317405]] (duration: 03m 23s)
* 19:37 dancy@deploy1003: Started scap sync-world: testing [[phab:T317405|T317405]]
* 19:36 dancy@deploy1003: Installation of scap version &quot;4.259.0&quot; completed for 2 hosts
* 19:34 dancy@deploy1003: Installing scap version &quot;4.259.0&quot; for 2 host(s)
* 18:55 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
* 18:55 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
* 18:43 jhathaway@cumin1003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Alangi Derick out of all services on: 2442 hosts
* 18:41 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2002
* 18:41 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2002
* 18:41 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2002
* 18:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2002.codfw.wmnet 50.16.192.10.in-addr.arpa 0.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:40 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2002.codfw.wmnet 50.16.192.10.in-addr.arpa 0.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 18:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:40 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2002 - herron@cumin1003&quot;
* 18:40 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2002 - herron@cumin1003&quot;
* 18:36 herron@cumin1003: START - Cookbook sre.dns.netbox
* 18:33 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2002
* 18:32 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2002.codfw.wmnet with OS trixie
* 18:26 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2003.codfw.wmnet with OS trixie
* 18:04 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage
* 18:00 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage
* 17:41 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2003
* 17:41 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2003
* 17:40 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2003
* 17:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2003.codfw.wmnet 24.32.192.10.in-addr.arpa 4.2.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:40 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2003.codfw.wmnet 24.32.192.10.in-addr.arpa 4.2.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:40 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:40 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2003 - herron@cumin1003&quot;
* 17:40 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2003 - herron@cumin1003&quot;
* 17:33 herron@cumin1003: START - Cookbook sre.dns.netbox
* 17:28 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2003
* 17:28 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2003.codfw.wmnet with OS trixie
* 17:15 herron@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2004.codfw.wmnet with OS trixie
* 16:34 cdobbins@cumin2002: conftool action : get/pooled; selector: name=cp5024.eqsin.wmnet
* 16:30 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 16:30 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 16:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
* 16:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
* 16:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2001.codfw.wmnet
* 15:59 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 15:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest2001.codfw.wmnet
* 15:47 dancy@deploy1003: Installation of scap version &quot;4.258.1&quot; completed for 2 hosts
* 15:45 dancy@deploy1003: Installing scap version &quot;4.258.1&quot; for 2 host(s)
* 15:34 herron@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2004.codfw.wmnet with reason: host reimage
* 15:30 herron@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2004.codfw.wmnet with reason: host reimage
* 15:14 herron@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2004
* 15:14 herron@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2004
* 15:11 herron@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2004
* 15:11 herron@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2004.codfw.wmnet 38.16.192.10.in-addr.arpa 8.3.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:11 herron@cumin1003: START - Cookbook sre.dns.wipe-cache kafka-logging2004.codfw.wmnet 38.16.192.10.in-addr.arpa 8.3.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 15:11 herron@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:11 herron@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2004 - herron@cumin1003&quot;
* 15:11 herron@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2004 - herron@cumin1003&quot;
* 15:05 dancy@deploy1003: Installation of scap version &quot;4.258.0&quot; completed for 2 hosts
* 15:03 dancy@deploy1003: Installing scap version &quot;4.258.0&quot; for 2 host(s)
* 14:57 herron@cumin1003: START - Cookbook sre.dns.netbox
* 14:47 herron@cumin1003: START - Cookbook sre.hosts.move-vlan for host kafka-logging2004
* 14:47 herron@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2004.codfw.wmnet with OS trixie
* 13:45 zabe@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
* 13:44 zabe@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
* 13:24 _Gerges: WikiMonitor setup
* 13:09 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080
* 13:09 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078
* 13:09 jclark@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1079
* 13:09 jclark@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1077
* 13:09 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080
* 13:09 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079
* 13:09 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078
* 13:08 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077
* 13:08 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:08 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:04 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:04 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1077 to eqiad - jclark@cumin1003&quot;
* 13:04 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1077 to eqiad - jclark@cumin1003&quot;
* 13:00 jclark@cumin1003: START - Cookbook sre.dns.netbox
* 12:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 12:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 12:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 12:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 09:57 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281423{{!}}Switch watchstar from Popover to Dialog (T417847)]] (duration: 06m 49s)
* 09:53 samtar@deploy1003: samtar: Continuing with deployment
* 09:52 samtar@deploy1003: samtar: Backport for [[gerrit:1281423{{!}}Switch watchstar from Popover to Dialog (T417847)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:50 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1281423{{!}}Switch watchstar from Popover to Dialog (T417847)]]
* 09:38 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1281426{{!}}Update the interwiki cache (T239173)]] (duration: 06m 05s)
* 09:32 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1281426{{!}}Update the interwiki cache (T239173)]]
* 08:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 08:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 08:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 03:26 akhatun@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 03:26 akhatun@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 00:16 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1280417{{!}}Add script to fix fr_deleted drifts (T424553)]] (duration: 07m 05s)
* 00:13 zabe@deploy1003: zabe: Continuing with deployment
* 00:11 zabe@deploy1003: zabe: Backport for [[gerrit:1280417{{!}}Add script to fix fr_deleted drifts (T424553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1280417{{!}}Add script to fix fr_deleted drifts (T424553)]]

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>2ahfzbccjfs6frtgb0ud6v0xansl8s8</sha1>
    </revision>
  </page>
  <page>
    <title>Release Engineering/SAL</title>
    <ns>0</ns>
    <id>17290</id>
    <revision>
      <id>2412191</id>
      <parentid>2411983</parentid>
      <timestamp>2026-05-09T20:46:24Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1285498</comment>
      <origin>2412191</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="43202" sha1="twcwwlc9zfq81wfsvmgqzavl8hlu79q" xml:space="preserve">=== 2026-05-09 ===
* 20:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1285498

=== 2026-05-07 ===
* 22:53 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/105

=== 2026-05-06 ===
* 18:13 bd808: Unblock 88.165.192.0/19
* 18:03 bd808: Unblock 94.208.0.0/14
* 17:56 bd808: Unblock 84.226.0.0/16
* 17:41 bd808: Unblock 94.34.0.0/16
* 17:35 bd808: Unblock 109.134.0.0/16

=== 2026-05-05 ===
* 21:20 James_F: Zuul: Provide Node 26 experimental jobs everywhere needed
* 21:04 James_F: Docker: Provide initial Node 26 images
* 19:01 James_F: Zuul: [mediawiki/extensions/PageAssessments] Add Scribunto dependency, for [[phab:T396135|T396135]]
* 14:58 dancy: rm /var/log/&lt;nowiki&gt;{&lt;/nowiki&gt;user.log.1,syslog.1,messages.1&lt;nowiki&gt;}&lt;/nowiki&gt; on deployment-eventgate-4.deployment- prep ([[phab:T425429|T425429]])

=== 2026-05-04 ===
* 15:19 dancy: Upgrading gitlab cloud runners (prod) from 1.35.1-do.3 to 1.35.1-do.5
* 14:51 dancy: Upgrading gitlab cloud runners (staging) from 1.35.1-do.3 to 1.35.1-do.5
* 10:40 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template

=== 2026-05-02 ===
* 20:49 James_F: Zuul: [mediawiki/core] Enforce PHP 8.4 &amp; 8.5 on release branches, all pass
* 19:27 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for MW release branches
* 19:19 James_F: Zuul: [mediawiki/extensions/BlogPage] Add dependencies
* 16:48 James_F: Hard-restarting Zuul to clear the huge number of i18n updates being re-submitted.
* 15:48 James_F: Zuul: [wikimedia-cz/*] Test in PHP 8.3+, dropping 8.2
* 14:02 TheresNoTime: Add bvibber to deployment-prep project
* 09:08 James_F: Docker: [quibble-*] Add php-luasandbox so we can test both modes in Scribunto

=== 2026-05-01 ===
* 15:42 James_F: Zuul: [wikimedia/lucene-explain-parser] Test in PHP 8.3+, dropping 8.2
* 15:42 James_F: Zuul: [wikimedia/textcat] Test in PHP 8.3+, dropping 8.2
* 15:42 James_F: Zuul: [mediawiki/tools/ParseWiki] Test in PHP 8.3+, dropping 8.2
* 15:42 James_F: zuul: Add ToprakM to CI allowlist
* 15:19 James_F: Zuul: [translatewiki] Test in PHP 8.3+, dropping 8.2
* 15:10 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add TestKitchen as a dependency, for [[phab:T425076|T425076]]
* 12:40 James_F: Zuul: [mediawiki/tools/code-utils] Test in PHP 8.3+, dropping 8.2
* 08:02 James_F: Zuul: Update xtex's e-mail in the allowlist
* 07:37 James_F: Zuul: Switch release branches' selenium jobs to PHP 8.3
* 07:33 James_F: Zuul: Test Wikimedia production libraries in PHP 8.3+, dropping 8.2

=== 2026-04-30 ===
* 21:36 brennen: gitlab-webhooks: building &amp; restarting to deploy https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40
* 20:26 James_F: Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting
* 20:16 James_F: jforrester@doc1004:~$ # sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn/ # [[phab:T415832|T415832]]
* 20:14 James_F: Zuul: [mediawiki/extensions/WebAuthn] Archive, for [[phab:T415832|T415832]] / [[phab:T303495|T303495]]
* 17:16 brennen: wikibugs: most maintainers at hackathon, so go release-engineering added as a maintainer while looking to debug error at https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/810904
* 15:19 mutante: upgrading zuul to 14.2.0-1 on &quot;new zuul&quot; machines ([[phab:T424879|T424879]])

=== 2026-04-29 ===
* 15:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add ConfirmEdit dependency, for [[phab:T424597|T424597]]
* 15:36 James_F: Zuul: Drop experimental node22 jobs, never used in practice
* 15:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1279392, https://gerrit.wikimedia.org/r/1279397

=== 2026-04-28 ===
* 18:11 bd808: Unblock 86.0.0.0/16
* 17:41 bd808: Unblock 79.192.0.0/10
* 17:07 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing
* 17:07 James_F: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing
* 17:07 James_F: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing
* 16:32 James_F: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing
* 13:34 James_F: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing
* 13:34 James_F: Zuul: [oojs/ui] Drop PHP 8.2 testing
* 13:14 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI
* 10:40 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud to fix failure seen in mwext-node24-rundoc #1717
* 00:03 bd808: Increase parallelism for wmf-beta-update-databases.py ([[phab:T256168|T256168]])

=== 2026-04-27 ===
* 22:11 bd808: Beta Cluster MediaWiki update logs now available via https://beta-update.wmcloud.org/ ([[phab:T256168|T256168]])
* 21:57 bd808: Add web security group to deployment-deploy04 ([[phab:T256168|T256168]])
* 20:45 James_F: Zuul: Restrict mw*-codehealth-patch jobs to master only, for [[phab:T424573|T424573]]
* 17:16 James_F: Docker: [mediawiki-phan-taint-check-demo] Re-platform to Trixie and so PHP 8.4
* 15:53 James_F: Zuul: [mediawiki/extensions/ReportIncident] Add TestKitchen phan dependency, for [[phab:T424220|T424220]]
* 14:32 James_F: Zuul: Drop PHP 8.2 enforcement from MediaWiki things for master and REL1_46 for [[phab:T358667|T358667]]
* 12:38 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 383
* 09:18 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # [[phab:T424419|T424419]]

=== 2026-04-26 ===
* 20:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276777

=== 2026-04-24 ===
* 22:48 dduvall: merged zuul3 branch of integration/config into master and pushed (in preparation for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277198)
* 12:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276428

=== 2026-04-23 ===
* 23:57 bd808: Set `profile::beta::autoupdater::run_updater: true` for deployment-deploy04 via Horizon ([[phab:T256168|T256168]])
* 22:58 bd808: bd808@deployment-deploy04 `sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-all`
* 22:36 bd808: bd808@deployment-deploy04 `sudo -u mwdeploy /usr/local/bin/wmf-beta-update-all`
* 22:16 bd808: Disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad so that replacement script can be tested ([[phab:T256168|T256168]])
* 22:12 bd808: Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad so that replacement script can be tested ([[phab:T256168|T256168]])
* 22:02 bd808: Cherry-picked {{gerrit|1276813}} to deployment-puppetserver-1 ([[phab:T256168|T256168]])
* 20:11 James_F: Zuul: [wikibase/*] Replace CI testing in Node 20 with Node 24
* 20:11 James_F: Zuul: [wikidata/query/*] Replace CI testing in Node 20 with Node 24
* 20:11 James_F: Zuul: [analytics/*] Replace CI testing in Node 20 with Node 24
* 20:10 James_F: Zuul: [mediawiki/tools/*] Replace CI testing in Node 20 with Node 24
* 20:06 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.34.5-do.3 to 1.35.1-do.3 ([[phab:T423726|T423726]])
* 19:55 James_F: Zuul: [jquery-client] Replace CI testing in Node 20 with Node 24
* 19:51 James_F: Zuul: [wikipeg] Drop testing in Node 20 and Node 22
* 19:47 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.34.5-do.3 to 1.35.1-do.3 ([[phab:T423726|T423726]])
* 19:37 James_F: Zuul: [oojs/ui] Drop CI testing in Node 20 and Node 22
* 19:37 James_F: Zuul: [oojs/js] Drop CI testing in Node 20 and Node 22
* 19:37 James_F: Zuul: [unicodejs] Replace CI testing in Node 20 with Node 24
* 19:36 James_F: Zuul: [wikimedia/portals] Drop CI testing in Node 20 and Node 22
* 18:57 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.33.9-do.3 to 1.34.5-do.3 ([[phab:T423726|T423726]])
* 18:39 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.3 to 1.34.5-do.3 ([[phab:T423726|T423726]])
* 18:18 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.2 to 1.33.9-do.3 ([[phab:T423726|T423726]])
* 17:58 James_F: Zuul: [mediawiki/extensions/OAuth] Add dependency on CentralAuth, for [[phab:T415281|T415281]]
* 17:56 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.13-do.2 to 1.33.9-do.3 ([[phab:T423726|T423726]])
* 16:35 James_F: Zuul: Enforce PHP 8.5 CI for MW things in master (and REL1_46), for [[phab:T411814|T411814]]
* 16:19 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.5 CI
* 15:47 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add AntiSpoof dependency, for [[phab:T420548|T420548]]
* 14:20 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure seen in mediawiki-node24 8385 and 8405
* 12:56 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add CentralNotice dependency, for [[phab:T422082|T422082]]

=== 2026-04-22 ===
* 00:07 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add MF dependency, for [[phab:T424113|T424113]]

=== 2026-04-21 ===
* 23:26 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too, for [[phab:T394410|T394410]]
* 23:17 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs, for [[phab:T422031|T422031]]
* 20:47 inflatador: updating cirrussearch hosts to Trixie/OpenSearch 2 [[phab:T421763|T421763]]
* 20:38 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep, for [[phab:T394410|T394410]]
* 20:17 bd808: Running tofu for [[phab:T421244|T421244]]
* 18:00 James_F: Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency
* 16:35 bd808: Unblock 79.116.0.0/16
* 13:34 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep, for [[phab:T415254|T415254]]
* 13:27 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add CentralAuth dependency, for [[phab:T420548|T420548]]

=== 2026-04-20 ===
* 23:56 bd808: Unblock 76.157.0.0/16
* 18:28 dancy: Upgrading gitlab cloud runners (staging) to 1.33.9-do.2 ([[phab:T423726|T423726]])
* 18:28 dancy: Upgrading gitlab cloud runners (staging) ([[phab:T423726|T423726]])
* 18:19 James_F: jjb: All 486 (!) jobs now updated for [[phab:T423622|T423622]]
* 18:18 bd808: Unblock 113.128.0.0/15
* 15:03 James_F: Docker: Bump ci-bullseye/-bookworm/-trixie for mirrors.wm.org removal, [[phab:T423622|T423622]]

=== 2026-04-19 ===
* 19:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272752

=== 2026-04-17 ===
* 21:07 thcipriani: marking integration-agent-1080 offline for experimentation
* 19:30 thcipriani: reconfiguring castor-save-workspace-cache with https://gerrit.wikimedia.org/r/1273935
* 17:47 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.10-do.1 to 1.32.13-do.2 ([[phab:T423726|T423726]])
* 16:49 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.32.10-do.1 to 1.32.13-do.2 ([[phab:T423726|T423726]])

=== 2026-04-16 ===
* 20:49 dduvall: creating integration/zuul-jobs repo to serve as a mirror of opendev.org/zuul/zuul-jobs ([[phab:T406384|T406384]])
* 13:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272711 [[phab:T423568|T423568]]
* 11:07 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud

=== 2026-04-15 ===
* 20:05 James_F: Zuul: Configure REL1_46 CI, for [[phab:T423257|T423257]]
* 17:44 bd808: Unblock 176.0.0.0/13
* 17:39 bd808: Unblock 46.128.0.0/16
* 17:32 bd808: Unblock 176.86.0.0/16
* 16:39 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/127d783b2176ac60b646a5fa4f1b1a872ca66340
* 15:33 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/100
* 01:02 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/99

=== 2026-04-14 ===
* 20:42 James_F: Docker: [composer-scratch] Upgrade composer to 2.9.7 and cascade
* 16:35 bd808: Unblock 88.112.0.0/14
* 00:48 bd808: Unblock 24.6.0.0/16
* 00:42 bd808: Unblock 152.231.48.0/20

=== 2026-04-13 ===
* 22:00 James_F: Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches
* 20:28 James_F: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used
* 14:42 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep, for [[phab:T421011|T421011]]

=== 2026-04-12 ===
* 18:21 James_F: jforrester@contint1002:~$ sudo /usr/sbin/service zuul restart &amp;&amp; tail -f -n100 /var/log/zuul/zuul.log # [[phab:T423027|T423027]]

=== 2026-04-10 ===
* 23:22 James_F: jforrester@contint1002:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/ReadingLists --change {{Gerrit|1269498}},2 # [[phab:T422976|T422976]]
* 23:20 James_F: Zuul: [mediawiki/extensions/ReadingLists] Publish JS coverage, for [[phab:T422976|T422976]]
* 23:13 James_F: Zuul: Migrate a few straggler Node 20 MediaWiki things to Node 24
* 23:01 James_F: Zuul: Move all MediaWiki things from mediawiki-node20 to mediawiki-node24
* 21:59 James_F: Docker: Bump Node base images to March releases and cascade; Upgrade Quibble images from Node 20 to Node 24
* 10:24 hashar: Updating all Quibble jobs to 1.17.1
* 10:22 hashar: Updated PostgreSQL jobs to Quibble 1.17.1 # [[phab:T422110|T422110]]
* 10:22 hashar: Updated apitesting job to Quibble 1.17.1 # [[phab:T422843|T422843]] [[phab:T418743|T418743]]
* 09:51 hashar: Tag Quibble 1.17.1 @ {{Gerrit|0a1ab3b7c3dfee36c9bc2e9b049957d94e190e85}}

=== 2026-04-09 ===
* 15:13 hashar: Rolling back Quibble jobs to 1.16.0 (api-testing stage fails due to missing npm install step`
* 14:58 hashar: Upgrading Quibble jobs to 1.17.0
* 14:23 hashar: Tagged Quibble 1.17.0 @ {{Gerrit|864381c6b63bdbcd8c74a3162c406fffcaaf8694}}
* 07:48 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1268559 &quot;Zuul: use standalone jobs for GrowthExperiments Cypress tests&quot; {{!}} [[phab:T417412|T417412]]

=== 2026-04-08 ===
* 22:19 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1269068
* 22:01 bd808: Unblock 95.216.12.170/32 ([[phab:T422751|T422751]])
* 19:26 brennen: gitlab-webhooks: building &amp; deploying https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/37 - hitting some build tooling stuff, trying a fix per instructions in the error log
* 17:54 bd808: Unblock 167.56.0.0/13 ([[phab:T422721|T422721]])
* 06:31 hashar: Deleted integration-agent-castor05 Bullseye instance, replaced by integration-agent-castor06 which is on Bookworm # [[phab:T421114|T421114]]
* 06:24 hashar: Deleted integration-agent-qemu-1003 Bullseye image, replaced by integration-agent-qemu-1004 which is on Bookworm # [[phab:T422488|T422488]]

=== 2026-04-07 ===
* 22:25 dduvall: adding new pipelinelib labels to ci nodes ([[phab:T422234|T422234]])
* 20:05 hashar: Triggered a build of https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/
* 17:06 dduvall: added `Docker` label to `contint` jenkins nodes ([[phab:T422507|T422507]])
* 17:05 dduvall: restored missing `pipelinelib` labels on `integration-agent-docker-` CI hosts ([[phab:T422507|T422507]])
* 16:53 bd808: Unblock 73.0.0.0/8 ([[phab:T422498|T422498]])
* 12:36 hashar: jjb: use $CASTOR_HOST for Quibble success cache. https://gerrit.wikimedia.org/r/1268545  {{!}}  This causes the Quibble jobs to use a new instance for the success cache, which is empty  # [[phab:T383243|T383243]]  [[phab:T421114|T421114]]
* 12:17 hashar: Migrated Castor from integration-castor05 to integration-castor06. Updated CASTOR_HOST in Jenkins and moved the Cinder volume to the new instance # [[phab:T421114|T421114]]
* 11:14 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1090, 1091, 1092 and 1093 # [[phab:T421114|T421114]]
* 10:09 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1083 to 1089 # [[phab:T421114|T421114]]
* 07:23 hashar: CI Jenkins: removed `blubber` label from all agents after having moved PipelineLib to use the `Docker` label {{!}} [[phab:T422234|T422234]]

=== 2026-04-06 ===
* 16:01 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1268239

=== 2026-04-03 ===
* 20:17 bd808: Unblock 2.54.0.0/16 ([[phab:T422238|T422238]])
* 17:25 bd808: Unblock 31.18.0.0/16 ([[phab:T422245|T422245]])
* 17:18 bd808: Unblock 2.54.128.0/19 ([[phab:T422238|T422238]])
* 16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1264649 &quot;add Python 3.14 to pywikibot jobs and separate lint tests&quot; {{!}} [[phab:T421723|T421723]]
* 09:26 hashar: integration: nuked pywikibot/core pre-commit cache # [[phab:T422242|T422242]]
* 09:15 hashar: Added Bookworm based Jenkins agents to the pool with label `Docker`. Hostnames are `integration-agent-docker-107*` # [[phab:T421114|T421114]]
* 02:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1267398

=== 2026-04-02 ===
* 16:50 thcipriani: restart jenkins
* 15:15 bd808: Unblock 82.216.0.0/16 ([[phab:T421508|T421508]])
* 15:07 bd808: Unblock 95.90.0.0/15 ([[phab:T421485|T421485]])
* 11:19 James_F: Zuul: [oojs/ui] Drop ooui-ruby2.7-rake job, we're abandoning Ruby use there

=== 2026-04-01 ===
* 22:01 bd808: Unblock 109.144.0.0/12 ([[phab:T422019|T422019]])
* 20:16 bd808: Unblock 93.192.0.0/10 ([[phab:T421894|T421894]])
* 19:25 dancy: Updating buildkitd to v0.29.0 in gitlab-cloud-runners (prod) ([[phab:T415284|T415284]])
* 17:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/97 ([[phab:T420441|T420441]])
* 17:39 bd808: Unblock 94.134.0.0/15 ([[phab:T421866|T421866]])
* 16:31 dancy: Upgrade buildkit to 0.29.0 in staging gitlab-cloud-runners ([[phab:T415284|T415284]])
* 10:47 taavi: integration-castor05: free up a bit of disk space by deleting cache for AhoCorasick/ CLDRPluralRuleParser/ HtmlFormatter/ RelPath/ RunningStat/ IPSet/

=== 2026-03-30 ===
* 22:01 bd808: Unblock 78.20.0.0/14 ([[phab:T421586|T421586]])
* 21:04 bd808: Unblock 95.88.0.0/15 ([[phab:T421774|T421774]])
* 20:49 bd808: Unblock 95.89.191.0/24 ([[phab:T421774|T421774]])
* 20:29 bd808: Unblock 73.162.0.0/16 ([[phab:T421549|T421549]])
* 13:10 hashar: gerrit: abandon mediawiki/core changes that are 2+years old and are attached to a task (`Bug: Txxxx`)
* 11:37 hashar: Reloaded Zuul to to add 3 persons to the  allow list
* 10:43 James_F: Docker: Re-pushing to try to create quibble-coverage 1.16.0-s2

=== 2026-03-27 ===
* 21:00 James_F: Docker: [quibble-bullseye] Drop Python 2 from images
* 11:28 hashar: deployment-prep: removed block for `143.176.0.0/15` and blocked subblock `143.176.0.0/16` instead. This unblocks `143.177.0.0/16` # [[phab:T421420|T421420]]
* 00:18 bd808: Unblock 95.90.238.0/23 ([[phab:T421447|T421447]])

=== 2026-03-26 ===
* 21:25 bd808: Unblock 89.240.0.0/15 ([[phab:T421364|T421364]])
* 21:09 brennen: patchdemo: deploy to production for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/312

=== 2026-03-25 ===
* 20:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256318 [[phab:T421283|T421283]]
* 15:46 dancy: Migrated gitlab-cloud-runners (prod) from nginx-ingress to traefik ([[phab:T420743|T420743]])
* 15:32 dancy: Migrated gitlab-cloud-runners (staging) from nginx-ingress to traefik ([[phab:T420743|T420743]])
* 10:01 hashar: Updating tox Jenkins jobs to add support for Python 3.14 {{!}} https://gerrit.wikimedia.org/r/1260632 {{!}} [[phab:T421209|T421209]]
* 08:40 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20/

=== 2026-03-24 ===
* 19:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255746
* 15:34 brennen: gitlab1004: manual test run of `configure-projects` with cleared issue allowlist ([[phab:T412882|T412882]])
* 15:26 bd808: Unblock 47.194.0.0/16 ([[phab:T421127|T421127]])
* 12:53 hashar: integration: deleted old Puppet 5 compiler agents from Jenkins ( pcc-worker1014.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1015.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1016.puppet-diffs.eqiad1.wikimedia.cloud ) # [[phab:T367399|T367399]]
* 07:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1259755

=== 2026-03-23 ===
* 15:28 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 90272

=== 2026-03-22 ===
* 14:52 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1258082
* 01:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256488

=== 2026-03-21 ===
* 08:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256962
* 07:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256946

=== 2026-03-20 ===
* 21:21 bd808: Unblock 103.159.218.0/24 ([[phab:T420530|T420530]])
* 14:59 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror, for [[phab:T399673|T399673]]

=== 2026-03-19 ===
* 16:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255777
* 16:01 Krinkle: Hoist l10n-bot rights from labs/tools parent to labs parent to reduce duplication in other labs/ repos
* 15:50 Krinkle: Create labs/xtools repo (branch: main, parent: labs, owner: labs-xtools), ref [[phab:T402086|T402086]]

=== 2026-03-18 ===
* 21:11 dcausse: [[phab:T403775|T403775]]: reindexing all wikis to enable new sorting options
* 21:08 dcausse: restarting opensearch on deployment-cirrussearch(12{{!}}13{{!}}14) instances to pickup new plugin versions
* 14:56 James_F: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest
* 14:52 James_F: Zuul: [GrowthExperiments] drop duplicate VisualEditor dep
* 14:52 James_F: Zuul: [search/*] Add experimental Java 25 jobs

=== 2026-03-17 ===
* 22:50 James_F: Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs
* 21:27 James_F: Zuul: search: Update opensearch plugins for Java 11/17, for [[phab:T420407|T420407]]
* 20:20 bd808: Resize deployment-sessionstore06 from g4.cores1.ram2.disk20 to g4.cores2.ram4.disk20 ([[phab:T415021|T415021]])
* 16:43 James_F: Zuul: [BlueSpicePermissionManager] Add …ConfigManager &amp; …UserManager deps
* 14:36 James_F: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep, for [[phab:T420015|T420015]]

=== 2026-03-13 ===
* 13:59 andrewbogott: deleting ptr record 117.0.16.172.in-addr.arpa. -- accidental duplicate for deployment-kafka-logging01.deployment-prep.eqiad1.wikimedia.cloud
* 13:04 elukey: re-create kafka-logging-01 in deployment-prep on trixie and Kafka 3.7 (was running on buster)
* 09:13 elukey: upgrade kafka-jumbo and kafka-main to Confluent 7.7 in deployment-prep (pre-requisite before being able to upgrade to Trixie)

=== 2026-03-12 ===
* 21:23 bd808: Hard reboot deployment-sessionstore06 ([[phab:T415021|T415021]])
* 01:14 James_F: Docker: [helm-linter] Bump for Envoy 1.35.9, for [[phab:T419637|T419637]]

=== 2026-03-11 ===
* 16:48 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/MetricsPlatform # [[phab:T417568|T417568]]
* 16:47 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Archive, for [[phab:T416865|T416865]]
* 11:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1250529 &quot;inference-services: Split policy violation CI into separate model jobs.&quot; - [[phab:T418832|T418832]]

=== 2026-03-10 ===
* 17:39 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner production
* 17:11 hashar: Updated MediaWiki coverage jobs so that they now keep &quot;Generate a local configuration by running `composer phpunit:config`&quot; message # [[phab:T419073|T419073]]
* 16:41 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner staging
* 08:21 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20

=== 2026-03-09 ===
* 21:53 bd808: Reboot deployment-shellbox01 on the off chance that is makes the new permissions error go away ([[phab:T419440|T419440]])
* 13:13 James_F: Zuul: [mediawiki/extensions/WikiShare] Mark as archived, for [[phab:T413589|T413589]]
* 13:11 James_F: Zuul: [mediawiki/extensions/Memento] Mark as archived, for [[phab:T369991|T369991]]
* 13:10 James_F: Zuul: [mediawiki/extensions/QuickGV] Mark as archived, for [[phab:T413348|T413348]]
* 13:10 James_F: Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived, for [[phab:T413588|T413588]]
* 13:09 James_F: Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived, for [[phab:T413587|T413587]]
* 13:07 James_F: Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived, for [[phab:T413586|T413586]]
* 13:06 James_F: Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived, for [[phab:T413585|T413585]]
* 13:04 James_F: Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived, for [[phab:T418008|T418008]]
* 12:50 James_F: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency
* 12:50 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen
* 10:40 hashar: gerrit: mediawiki/vendor: converted `es6` and `es710` branches to tags # [[phab:T417804|T417804]]
* 09:24 hashar: Updating Quibble jobs to 1.16.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1248880 {{!}} [[phab:T417399|T417399]] [[phab:T417409|T417409]] [[phab:T418461|T418461]]
* 09:15 hashar: updating all CI Jenkins jobs using `./jjb-update`

=== 2026-03-06 ===
* 19:46 James_F: Zuul: [mediawiki/services/geoshapes] Mark as archived, for [[phab:T418372|T418372]]
* 16:37 hashar: Building Docker images for Quibble 1.16.0
* 16:31 hashar: Tag Quibble 1.16.0 @ {{Gerrit|0b9db5fe3cabb2cec0b5d44e128bafa917b3b895}} # [[phab:T417399|T417399]] [[phab:T417409|T417409]] [[phab:T418461|T418461]]
* 12:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248411 &quot;jjb, Zuul: vary Wikibase Selenium for release branches&quot; {{!}} [[phab:T418797|T418797]]
* 12:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248409/ &quot;jjb, Zuul: rename wikibase-selenium job for clarity&quot; {{!}} [[phab:T418797|T418797]]

=== 2026-03-05 ===
* 14:41 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add TestKitchen as a dependency for [[phab:T418053|T418053]]
* 08:01 hashar: Reloaded Zuul to rename wikibase-client / wikibase-repo jobs {{!}} https://gerrit.wikimedia.org/r/1238317
* 00:04 James_F: Docker: [quibble-coverage] Use local PHPUnit config, for [[phab:T345481|T345481]]

=== 2026-03-04 ===
* 21:16 James_F: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch, for [[phab:T411814|T411814]]
* 21:10 James_F: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch, for [[phab:T411814|T411814]]
* 19:48 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 ([[phab:T419004|T419004]])
* 18:50 James_F: Revert &quot;Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency&quot;, for [[phab:T419043|T419043]]
* 16:23 James_F: Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting
* 15:37 James_F: Docker: [rake-ruby2.7] Add libffi-dev too, for [[phab:T418463|T418463]]
* 13:59 James_F: Docker: [rake-ruby2.7] Add ruby-ffi for [[phab:T418463|T418463]]
* 13:54 hashar: SIGKILL Zuul cause it can't gracefully stop most probably due to being locked attempting to report back to Gerrit # [[phab:T419009|T419009]]
* 13:49 hashar: Stopping Zuul # [[phab:T419009|T419009]]
* 13:41 hashar: Took a Zuul stack dump on contint1002.wikimedia.org using SIGUSR1 # [[phab:T419009|T419009]]

=== 2026-03-03 ===
* 23:52 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Drop MetricsPlatform phan dep
* 23:52 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Drop MetricsPlatform phan dep

=== 2026-03-02 ===
* 22:13 James_F: Zuul: Enforce PHP 8.4 in MW extensions and skins for development branch, for [[phab:T386108|T386108]]
* 14:05 James_F: Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency, for [[phab:T415451|T415451]]
* 13:48 James_F: Zuul: […/WikimediaEvents] Drop LoginNotify dependency, now unused, for [[phab:T404334|T404334]]
* 10:16 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/15.8.2/ # [[phab:T418718|T418718]]

=== 2026-02-28 ===
* 21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/skins`  # [[phab:T418675|T418675]]
* 21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/extensions`  # [[phab:T418675|T418675]]

=== 2026-02-27 ===
* 15:53 dancy: Updating gitlab-cloud-runners (staging and prod) to gitlab-runner 18.9.0.

=== 2026-02-26 ===
* 20:16 James_F: Zuul: Provide a custom, high-priority pipeline just for puppet compiler [[phab:T414621|T414621]]
* 19:32 James_F: Docker: Bump all the PHPs.
* 13:40 hashar: Deployed Jenkins job https://integration.wikimedia.org/ci/job/wikibase-selenium/ # [[phab:T287582|T287582]]
* 00:13 dduvall: forcing replacement of buildkitd helm release in gitlab-cloud-runner prod cluster due to dependency on removed k8s secret ([[phab:T416260|T416260]])

=== 2026-02-25 ===
* 23:50 dduvall: deploying https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 to gitlab-cloud-runner production cluster ([[phab:T416260|T416260]])
* 14:07 James_F: Zuul: [mediawiki/extensions/CommunityRequests] Add TemplateData dependency, for [[phab:T401638|T401638]]
* 00:08 jeena: no-op testing updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95

=== 2026-02-24 ===
* 15:55 brennen: devtools: test deploy phab/phorge to test instance ([[phab:T418256|T418256]])

=== 2026-02-23 ===
* 23:07 jeena: Updated development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92
* 22:43 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92
* 22:12 bd808: Unblock 191.80.192.0/18 ([[phab:T418132|T418132]])
* 20:26 hashar: Deleted &quot;replication-upstream&quot; Grafana dashboard in favor of a copy/new &quot;replication&quot; one.  https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream , replaced it by https://grafana.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4ebb/replication
* 16:29 James_F: Zuul: [3d2png] Add basic Node CI at version 20

=== 2026-02-20 ===
* 21:47 bd808: Unblock 168.184.84.0/24 ([[phab:T418020|T418020]])
* 17:13 bd808: Unblock 122.187.64.0/18 ([[phab:T417964|T417964]])
* 14:35 James_F: Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section

=== 2026-02-19 ===
* 18:34 bd808: Unblock 181.98.0.0/16 ([[phab:T417890|T417890]])
* 17:21 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Add AbuseFilter as a dependency, for [[phab:T417799|T417799]]
* 13:22 hashar: Reloaded Zuul to archive the Cergen repository {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1240688 {{!}} [[phab:T417887|T417887]]

=== 2026-02-18 ===
* 20:17 jeena: Updating development images on contint primary for [[phab:T415922|T415922]]
* 19:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240360
* 18:40 bd808: Unblock 46.59.0.0/17 ([[phab:T417747|T417747]])
* 17:05 hashar: Regenerating Jenkins jobs with JJB based on https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/
* 17:04 hashar: Added EXT_DEPENDENCIES to Quibble Jenkins jobs parameters so we can manually trigger them from the Web UI using a different set of deps # https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/
* 16:30 hashar: Triggered https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ with empty Zuul parameters introduced by https://gerrit.wikimedia.org/r/1240333 {{!}} https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4893/console
* 15:43 James_F: Zuul: [mediawiki/extensions/ReadingLists] Add EventBus dependency for [[phab:T417706|T417706]]
* 12:15 hashar: zuul-1001.zuul3.eqiad1.wikimedia.cloud: added keepalive=20 to the scheduler Gerrit driver and restarted scheduler container # [[phab:T417497|T417497]]
* 06:58 jeena: Updating development images on contint primary for [[phab:T415922|T415922]]

=== 2026-02-17 ===
* 23:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240081
* 23:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240078
* 15:58 brennen: deployed latest phab/phorge wmf/stable to devtools test instance ([[phab:T417657|T417657]])
* 09:01 hashar: Reloaded Zuul to enable php 8.5 testing on utfnormal, php-session-serializer, wikipeg, mediawiki/libs/Dodo, mediawiki/libs/UUID, testing-access-wrapper and translatewiki # [[phab:T406326|T406326]]

=== 2026-02-16 ===
* 15:27 hashar: Manually cleaned some old workspaces on integration-agent-docker-1042

=== 2026-02-12 ===
* 20:07 James_F: Zuul: Enable PHP 8.5 jobs for most MW libraries, for [[phab:T406326|T406326]]
* 19:33 James_F: Docker: [php83] Re-build with upstream's new 8.3.30 release and cascade
* 19:31 James_F: Zuul: Add PHP 8.5 CI job to various things noted as blocked by Phan, for [[phab:T410941|T410941]], [[phab:T406326|T406326]]
* 16:35 Krinkle: Disable publishing noise on tasks from repos Bcp47, clover-diff, ScopedCallback, and IDLeDOM. Ref [[phab:T143162|T143162]]
* 15:53 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/87
* 11:21 James_F: Zuul: [mediawiki/libs/shellbox] Add direct Phan job, for [[phab:T416064|T416064]]

=== 2026-02-10 ===
* 20:16 dancy: Rebooted k3s.catalyst-dev (it was unresponsive, but the reboot hasn't helped)

=== 2026-02-09 ===
* 21:58 James_F: Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job, for [[phab:T410941|T410941]]
* 19:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1238006 [[phab:T415680|T415680]]
* 11:51 James_F: Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency, for [[phab:T414435|T414435]]

=== 2026-02-05 ===
* 17:58 James_F: Zuul: […/WikimediaCustomizations] Add six new dependencies for [[phab:T404334|T404334]]
* 15:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1237254
* 15:18 James_F: Zuul: […/OATHAuth] Add dependency and phan dependency on CentralAuth

=== 2026-02-04 ===
* 12:54 James_F: Zuul: [mediawiki/extensions/Petition] Add CLDR dependency
* 10:03 hashar: Restarted Jenkins on releases2003.codfw.wmnet

=== 2026-02-02 ===
* 21:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1234926 &quot;re-enable master jobs for some BlueSpice repos - [[phab:T403196|T403196]]&quot;
* 21:05 bd808: Unblock 85.146.0.0/17 ([[phab:T416079|T416079]])
* 19:47 James_F: Zuul: […/WikimediaCustomizations] Add cldr phan dependency, for [[phab:T404334|T404334]]
* 17:33 bd808: Unblock 188.188.0.0/15 ([[phab:T416095|T416095]])
* 17:26 bd808: Unblock 85.94.84.0/22 ([[phab:T416105|T416105]])
* 17:09 bd808: Unblock 94.234.0.0/16 ([[phab:T416165|T416165]])
* 16:51 dancy: Update gitlab-runners to alpine-v18.6.6 ([[phab:T415214|T415214]])
* 16:27 bd808: Unblock 47.231.208.0/21 ([[phab:T416010|T416010]])
* 11:39 James_F: Zuul: […/WikimediaCustomizations] Add five new phan dependencies, for [[phab:T404334|T404334]]
* 09:45 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 58532, 58557

=== 2026-01-31 ===
* 21:49 James_F: Deleted Jenkins's job entry for castor-save-workspace-cache {{Gerrit|6193776}} and this seems to have unstuck things for [[phab:T416078|T416078]]?
* 21:45 James_F: Running `sudo systemctl restart jenkins` on contint for [[phab:T416078|T416078]]
* 21:44 James_F: Fighting [[phab:T416078|T416078]], took integration-castor-5 offline, disconnected, sshed in to kill threads, then reconnected; no change in aspect.
* 19:03 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1235380

=== 2026-01-28 ===
* 21:26 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn # [[phab:T415832|T415832]]
* 21:11 bd808: Unblock 181.160.0.0/15 &amp; 186.40.128.0/17 ([[phab:T415820|T415820]])
* 17:01 bd808: Unblock 102.182.0.0/16 ([[phab:T415782|T415782]])

=== 2026-01-27 ===
* 16:45 James_F: Zuul: Switch skin-quibble template with identical extension-quibble, for [[phab:T402398|T402398]]
* 16:18 James_F: Zuul: [ArticleGuidance] mention it will be in production
* 15:55 James_F: Docker: [quibble-bullseye] Update to Quibble 1.15.0
* 15:12 James_F: Docker: [quibble-coverage] Pass PHPUnit config location explicitly, for [[phab:T395470|T395470]]
* 09:18 hashar: integration: on integration-castor05, deleted caches for old MediaWiki branches
* 09:15 hashar: integration: on pkgbuilder instances, removed Buster cow images, aptcache and hooks. `sudo cumin --force -p 0 'name:pkgbuilder' 'rm -fR /srv/pbuilder/&lt;nowiki&gt;{&lt;/nowiki&gt;base-buster-amd64.cow,hooks/buster,aptcache/buster-amd64&lt;nowiki&gt;}&lt;/nowiki&gt;'`  # [[phab:T397209|T397209]]
* 09:14 hashar: integration: cleaned up old workspaces under /srv/jenkins/workspace

=== 2026-01-26 ===
* 23:27 bd808: Unblock 66.130.0.0/15 ([[phab:T415596|T415596]])
* 22:52 bd808: Unblock 45.16.0.0/12 ([[phab:T415467|T415467]])
* 14:46 hashar: gerrit: changed `operations/software/permissions` project type from `CODE` to `PERMISSIONS` by pointing `HEAD` to `refs/meta/config`

=== 2026-01-22 ===
* 17:36 James_F: Docker:  [quibble-coverage] Stop using legacy PHPUnit entrypoint ([[phab:T395470|T395470]]) &amp; Stop excluding Dump/ParserFuzz/Stub groups ([[phab:T415230|T415230]])
* 15:11 James_F: Zuul: [mediawiki/extensions/Math] Add a standalone job, for [[phab:T415230|T415230]]

=== 2026-01-20 ===
* 20:38 bd808: Cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1229186 ([[phab:T415113|T415113]])
* 19:05 bd808: Rebooted deployment-cache-text08 to see if the mystery haproxy startup failure would go away ([[phab:T415100|T415100]])
* 18:50 bd808: Unblock 152.7.0.0/16 ([[phab:T415100|T415100]])

=== 2026-01-17 ===
* 23:32 ori: beta-scap with `php_l10n: true` completed successfully: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/241466/console. PHP l10n files generated. Reverted local change to scap.cfg.
* 23:26 ori: Temporarily set `php_l10n: true` on deployment-deploy04:/etc/scap.cfg to see if next scap succeeds.

=== 2026-01-16 ===
* 16:33 dancy: Deleting deployment-mx03.deployment-prep ([[phab:T412975|T412975]])

=== 2026-01-15 ===
* 14:50 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/ArticleSummaries/ # [[phab:T413232|T413232]]

=== 2026-01-14 ===
* 17:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226907
* 16:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226893
* 15:57 bd808: Unblock 190.60.63.0/24 ([[phab:T414541|T414541]])

=== 2026-01-13 ===
* 15:04 James_F: Zuul: Make quibble-for-mediawiki-core-vendor-mysql-php84 voting, for [[phab:T386108|T386108]]

=== 2026-01-12 ===
* 21:33 zabe: zabe@deployment-mwmaint03:~$ foreachwiki migrateLinksTable.php --table imagelinks # [[phab:T413668|T413668]]
* 21:06 bd808: Unblock 66.81.168.0/21 ([[phab:T414303|T414303]])
* 17:42 dancy: Turned off instance deployment-prep.deployment-mx03
* 11:44 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 46331, 46344

=== 2026-01-10 ===
* 21:48 taavi: reload zuul for https://gerrit.wikimedia.org/r/1224782
* 00:25 bd808: Unblock 91.160.0.0/12 ([[phab:T414190|T414190]])

=== 2026-01-09 ===
* 17:33 thcipriani: re-enabling beta update jobs after test bad extension-list [[phab:T411516|T411516]]
* 17:09 thcipriani: disabling beta update jobs to test bad extension-list [[phab:T411516|T411516]])

=== 2026-01-08 ===
* 21:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224815 [[phab:T414136|T414136]]
* 18:24 bd808: Unblock 89.80.0.0/12 ([[phab:T414113|T414113]])
* 15:55 dancy: Upgrading gitlab-runner to v18.5.0 on gitlab-cloud-runners. ([[phab:T414053|T414053]])

=== 2026-01-07 ===
* 23:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1082574 https://gerrit.wikimedia.org/r/1224157 https://gerrit.wikimedia.org/r/1224159
* 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896311 [[phab:T27482|T27482]]
* 23:06 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224218
* 17:34 James_F: Zuul: Add new extensions: IssueTrackerLinks, PreviewLinks, and WikiRAG
* 17:34 James_F: Zuul: [labs/tools/heritage] Point to the task to drop 8.1 testing
* 15:09 James_F: Zuul: [labs/tools/heritage] Add testing in PHP 8.2+, not just PHP 8.1
* 15:03 James_F: Zuul: Even for extension-broken, don't offer PHP 8.1 testing
* 15:02 James_F: Zuul: Move quibble experimental sqlite/postgres tests to PHP 8.3

=== 2026-01-06 ===
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223690 [[phab:T411814|T411814]]
* 16:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223189 [[phab:T411814|T411814]]
* 00:30 bd808: Unblock 85.134.128.0/17 ([[phab:T413755|T413755]])
* 00:02 bd808: Unblock 89.166.128.0/17 ([[phab:T413702|T413702]])

=== 2026-01-05 ===
* 23:57 bd808: Unblock 185.233.104.0/22 ([[phab:T413472|T413472]])
* 23:51 bd808: Unblock 45.62.112.0/21 ([[phab:T413079|T413079]])
* 23:44 bd808: Unblock 85.134.200.0/21 ([[phab:T413067|T413067]])
* 19:03 dancy: Updated buildkitd to v0.26.3 in gitlab-cloud-runners
* 14:27 taavi: reload zuul for {{Gerrit|1223191}}
* 13:57 James_F: Zuul: [mediawiki/php/wmerrors] Enable PHP 8.5 testing, for [[phab:T410921|T410921]]

=== 2026-01-03 ===
* 17:59 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1222709 https://gerrit.wikimedia.org/r/1220388 https://gerrit.wikimedia.org/r/1219140

=== 2026-01-02 ===
* 17:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1222597

=== 2026-01-01 ===
* 02:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1221644

&lt;noinclude&gt;'''Server Admin Log''' logged from {{IRC|wikimedia-releng}} for [[Nova Resource:Deployment-prep|Beta Cluster]], [[mw:Continuous integration|Continuous integration]] and various other Release Engineering projects.&lt;/noinclude&gt;

{{SAL-archives/Release Engineering}}

&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>twcwwlc9zfq81wfsvmgqzavl8hlu79q</sha1>
    </revision>
  </page>
  <page>
    <title>User:Tommy Kronkvist</title>
    <ns>2</ns>
    <id>441756</id>
    <revision>
      <id>2412197</id>
      <parentid>2243746</parentid>
      <timestamp>2026-05-10T09:37:28Z</timestamp>
      <contributor>
        <username>Tommy Kronkvist</username>
        <id>11506</id>
      </contributor>
      <comment>User statistics.</comment>
      <origin>2412197</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="2084" sha1="mvq82ge50nzfc30zmddc3yb6vza5nbn" xml:space="preserve">Registered Wikimedia user since March 29, 2005,&lt;small&gt;&lt;sup&gt;(&lt;span class=&quot;plainlinks&quot;&gt;[https://sv.wikipedia.org/w/index.php?title=Special:Bidrag&amp;offset=20050329034322&amp;limit=1&amp;contribs=user&amp;target=Tommy+Kronkvist&amp;namespace=&amp;tagfilter=&amp;start=&amp;end=2005-03-29 verify]&lt;/span&gt;)&lt;/sup&gt;&lt;/small&gt; and a Wikispecies bureaucrat since 2015 plus interface administrator since 2019.&lt;small&gt;&lt;sup&gt;(&lt;span class=&quot;plainlinks&quot;&gt;[https://species.wikimedia.org/w/index.php?title=Special:ListUsers&amp;limit=1&amp;username=Tommy_Kronkvist verify]&lt;/span&gt;)&lt;/sup&gt;&lt;/small&gt; My global account information for all of Wikimedia [[meta::Special:CentralAuth/Tommy_Kronkvist|can be found here]], including detailed user statistics of my circa 392,100 Wikimedia edits (per May 10, 2026).

I'm also registered at Phabricator&lt;small&gt;&lt;sup&gt;(&lt;span class=&quot;plainlinks&quot;&gt;[https://phabricator.wikimedia.org/p/Tommy_Kronkvist/ user page]&lt;/span&gt;)&lt;/sup&gt;&lt;/small&gt; and administer the official &lt;span class=&quot;plainlinks&quot;&gt;[https://twitter.com/Wikispecies @Wikispecies]&lt;/span&gt; X account (formerly known as Twitter).

{{clear|all}}
&lt;br&gt;
{| class=&quot;toccolours&quot; style=&quot;width: 100%; cellpadding=4;&quot;
|- 
| width=&quot;300&quot; style=&quot;background: #ccf; text-align: right; padding-top: 0.3em; padding-right: 0.4em; padding-bottom: 0.3em; width: 15%; font-size: 0.8em; font-weight: bold;&quot; | ORCID identifier: 
| &amp;nbsp; [https://orcid.org/0000-0001-9968-1747 0000-0001-9968-1747]
|-
| width=&quot;300&quot; style=&quot;background: #ccf; text-align: right; padding-top: 0.3em; padding-right: 0.4em; padding-bottom: 0.3em; width: 15%; font-size: 0.8em; font-weight:bold;&quot; | Public [[:w:Pretty Good Privacy|PGP]] key: 
| &amp;nbsp; [[species:User:Tommy Kronkvist/PGP|19035D3D]]
|-
|-
| width=&quot;300&quot; style=&quot;background: #ccf; text-align: right; padding-top: 0.3em; padding-right: 0.4em; padding-bottom: 0.3em; width: 15%; font-size: 0.8em; font-weight:bold;&quot; | [[meta:Template:User committed identity|Committed identity]] hash: 
| style=&quot; font-size: 0.8em;&quot; | &amp;nbsp; a6edd6d2fdbf82621f0cda4e5525c71f8da9b5dfd308242c3c63365e998c32c5406b75448380903265a5403edffd1a0435b61ac943f3c65870db9250f8b884a9
|}</text>
      <sha1>mvq82ge50nzfc30zmddc3yb6vza5nbn</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2412192</id>
      <parentid>2412168</parentid>
      <timestamp>2026-05-10T00:01:21Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2412192</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1480" sha1="0iq4sl36xa9ke5x464n8ujgagi4oqfg" xml:space="preserve">{{/Header}}
== Today (2026-05-10) ==
== Yesterday (2026-05-09) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| es7 || [[phab:T419961|Login (T419961)]] (fceratto)
|-
| s3 || 
* [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
* [[phab:T425318|Switchover s3 master (db1189 -&amp;gt; db1223) (T425318)]] (marostegui)
|-
| s6 || [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
|-
| s7 || [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
|-
| s8 || [[phab:T419961|Login (T419961)]] (fceratto)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| ms2 || [[phab:T418979|Productionize db225[0-3] (T418979)]] (marostegui)
|-
| s2 || 
* [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
* [[phab:T424848|Switchover s2 master (db2207 -&amp;gt; db2204) (T424848)]] (marostegui)
|-
| s3 || 
* [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
* [[phab:T424864|Switchover s3 master (db2209 -&amp;gt; db2205) (T424864)]] (marostegui)
|-
| s4 || [[phab:T419961|Login (T419961)]] (fceratto)
|-
| s5 || [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
|-
| s6 || [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>0iq4sl36xa9ke5x464n8ujgagi4oqfg</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:Gitlab-account-approval/Log</title>
    <ns>116</ns>
    <id>453906</id>
    <revision>
      <id>2412189</id>
      <parentid>2412060</parentid>
      <timestamp>2026-05-09T13:45:30Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>bwiki was rejected.</comment>
      <origin>2412189</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="53549" sha1="0pku9p4zy2qad9abfz1wb1vpr8acqrv" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>0pku9p4zy2qad9abfz1wb1vpr8acqrv</sha1>
    </revision>
    <revision>
      <id>2412196</id>
      <parentid>2412189</parentid>
      <timestamp>2026-05-10T05:12:12Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>wobadha was rejected.</comment>
      <origin>2412196</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="53642" sha1="ax7x6zb4on6tjy71zd89m2r394k19w4" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-05-10 ===
* 05:12 &quot;wobadha&quot; was rejected (pending since 2026-02-08T05:11:00.569Z).

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>ax7x6zb4on6tjy71zd89m2r394k19w4</sha1>
    </revision>
    <revision>
      <id>2412198</id>
      <parentid>2412196</parentid>
      <timestamp>2026-05-10T11:09:13Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>yalihupokn was rejected.</comment>
      <origin>2412198</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="53718" sha1="rkof6a172k1fbpb3srmd2witzduc4rv" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-05-10 ===
* 11:09 &quot;yalihupokn&quot; was rejected (pending since 2026-02-08T11:06:51.336Z).
* 05:12 &quot;wobadha&quot; was rejected (pending since 2026-02-08T05:11:00.569Z).

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>rkof6a172k1fbpb3srmd2witzduc4rv</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.matchandsplit/SAL</title>
    <ns>498</ns>
    <id>454648</id>
    <revision>
      <id>2412195</id>
      <parentid>2205848</parentid>
      <timestamp>2026-05-10T04:03:10Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~soda@tools-bastion-15: ./matchandsplit/scripts/toolforge-deploy-new-version.sh</comment>
      <origin>2412195</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1007" sha1="6hix0gmjv4qy7z5ju7fefxn5nojnogk" xml:space="preserve">=== 2026-05-10 ===
* 04:03 wmbot~soda@tools-bastion-15: ./matchandsplit/scripts/toolforge-deploy-new-version.sh

=== 2024-07-14 ===
* 18:13 wmbot~soda@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh
* 18:02 wmbot~soda@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh
* 18:00 wmbot~soda@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh

=== 2024-06-24 ===
* 18:39 wmbot~soda@tools-bastion-13: Test
* 17:04 wmbot~theprotonade@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh
* 16:47 wmbot~theprotonade@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh

=== 2024-06-19 ===
* 11:20 wmbot~soda@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh
* 11:12 wmbot~soda@tools-bastion-13: Use newer OAuth credentials

=== 2024-06-13 ===
* 17:56 wmbot~soda@tools-bastion-13: ./matchandsplit/scripts/toolforge-deploy-new-version.sh


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>6hix0gmjv4qy7z5ju7fefxn5nojnogk</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-monitoring/SAL</title>
    <ns>498</ns>
    <id>459243</id>
    <revision>
      <id>2412190</id>
      <parentid>2411360</parentid>
      <timestamp>2026-05-09T17:49:59Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)</comment>
      <origin>2412190</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="12632" sha1="rcxjkbevrloaf67t2dr0j6ymfbtbnkg" xml:space="preserve">=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>rcxjkbevrloaf67t2dr0j6ymfbtbnkg</sha1>
    </revision>
  </page>
</mediawiki>